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ABSTRACT 



The Department of Education's Web site (http://www.ed.gov) 
is a rich information and service resource that is widely used both within 
the Department of Education and by a variety of users external to the 
Department. It provides extensive information about the Department and its 
programs and services, enables users to interact efficiently with some 
programs, and serves diverse audiences. As the Web site has grown and 
developed, a number of issues have arisen that require attention. This 
document presents results of an evaluation of the site from four 
perspectives: management assessment, policy analysis, log and transaction 
analysis, and usability testing. The full report is presented in five major 
sections and capped by an Executive Summary. Following Section I: 
"Introduction” in which the study objectives and methods are outlined, each 
of the four assessment techniques and associated findings are represented as 
individual sections- -Section II: "Management Assessment"; Section III: 

"Policy Assessment"; Section IV: "Technical Assessment"; and Section V: "User 
Assessment." Within each section, the methodology is detailed, findings 
presented, and recommendations suggested. Appendices include details of data 
collection instruments and other background information. (Author/AEF) 
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TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



EXECUTIVE SUMMARY 



The Department of Education’s website, www.ed.gov . is a rich information and 
service resource that is widely used both within the Department and by a variety of users 
external to the Department. It provides extensive information about the Department and 
its programs and services, enables users to interact efficiently with some progr^s (e.g., 
student financial aid), and serves diverse audiences. It has a rich web policy environment 
to facilitate Departmental decision-making in support of the development of the website. 
In relationship to other Federal websites, it is extremely well developed with a set of 
policies that go far beyond what other agencies typically have developed. But as the 
website has grown and developed, a number of issues - especially related to management 
and resource support — require attention. 

In recent years. Congress and the Administration have instigated numerous 
programs and initiatives in electronic information dissemination and in programmatic 
areas for which the department has responsibilities for implementation and oversight. In 
this context of significant growth in departmental responsibilities, there have been major 
demands to both respond to these responsibilities and to provide information and program 
services via the department’s website. The department’s efforts to move to a web-based 
environment is laudable, but it simply cannot maintain this effort without a reassessment 
of resource support and management structure - especially given the increased program 
initiatives required by congress and the administration. 

The Department contracted with the study team to evaluate the site during Fall 
1998. The team, with extensive experience in evaluation of Federal websites, performed 
an evaluation that examined the site from four perspectives: 

• Management assessment: This approach begins with a management 
perspective of how the agency is organized to manage Departmental websites 
in terms of developing content, adding and deleting pages, controlling quality, 
responding to customers, and evaluating the website (among other factors). 

• Policy analysis: This approach stresses the degree to which the Department 
has developed internal policies to manage the website and the degree to which 
those policies are understandable and implemented. This approach also 
considers the degree to which existing agency policies do or do not recognize 
government-wide policies dealing with such topics as records management, 
privacy. Freedom of Information (FOI), security. Government Performance 
Results Act (GPRA), etc. 

• Log and transaction analysis: This approach includes a range of assessment 
techniques that rely on data contained in the web server. Some of these 
include analyses of user paths, user search commands, and frequencv of page 
access. 
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• Usability testing; This approach assesses the usability of the website from the 
perspective of users. Typically user? examine the website in terms of 
content, ease of use, navigability, organization, availability and quality of 
help, etc. 



The study team gathered data and identified key issues in the four areas. These are 
detailed in the full report and synthesized here. Additionally, the study team developed a 
set of recommendations for each area and a set of general recommendations that span the 
four areas. Some of these recommendations are in conflict with each other as can be 
expected given the range of stakeholders for the site. The Department will need to 
exercise judgement in determining how best to utilize the recommendations. 

The team’s evaluation provides a snapshot of www.ed.gov at this moment in time. 
As such it may under-represent the historical context which has led to the site’s current 
status. It should also not be construed to represent the only large-scale evaluation that 
should occur for the site. 



OVERVIEW OF THE REPORT 

The full report is presented in five major sections and capped by the summary that 
is presented here. Following an introduction (Section I) in which the study objectives 
and methods are outlined, each of the four assessment techniques and associated findings 
are represented as individual sections (Management Assessment: section II; Policy 
Assessment: section III; Technical Assessment; section IV; and User Assessment: 
Section V.) Within each section, the methodology is detailed, findings presented, and 
recommendations suggested. In conjunction with the report, an extensive selection of 
appendices is provided which include details of data collection instruments and other 
background information. Finally, the study team provided the department with two 
supplemental loose-leaf notebooks of sets of policy instruments, one of all ED website 
policies, and the other of representative website policies from other federal agencies. 



KEY FINDINGS 

As stated earlier, the Department has quite successfully “grown” its website creating 
a rich information environment within a rich policy and management context. However, 
the Department and the website are following a not untypical trajectory for organizational 
websites, a trajectory that, if continued, will have a number of negative consequences for 
the website. This trajectory is one in which resources and management structures do not 



keep pace with the demands made on the website and the staff associated with it. 

Most critically, the ED website: 

• Lacks adequate staffing and resource support for current operations and this condition 
is likely to worsen dramatically as demands for Internet support grow. 

• Is outgrowing the capacity of the current website management structure. 

These two issues represent “make-or-break” issues for the website. Without additional 

resources and a reconsideration of the current management structure, the problems 

currently experienced will continue and likely grow worse. 

In addition to these two critical problems, the website: 

• Needs to have expanded ongoing evaluation efforts associated with its use and 
growth. 

• May be attracting users who are not well served by the website structure, 
organization, or associated finding tools and services. 

• Is demonstrating some technical problems (such as non-standard coding, insufficient 
usage of META tags) which hinder user access. 

• Can better incorporate and comply with selected broader Federal policies and 
regulations. 

• Needs to move content beyond the current situation of reusing print content to a 
customer-centric site, which may present different information in ways that take 
advantage of the web’s capabilities. 

These conclusions result from the study team’s four-pronged analysis approach. 



MANAGEMENT AND EVALUATION EFFORTS 

The most pressing concern for the Department regarding its website is the 
imminent outstripping of the capacity of the management structure and resources to meet 
the demands of the website. The team’s evaluation provides a picture of a structure and 
resource base severely strained by the current demands of the website (see Figure ES -1). 
As the Department can expect these demands to continue growing, it is obvious that 
problems identified in this report will be exacerbated and likely other problems will 
emerge. 



Figure ES-1 Web-Based Service 
Demands VS. Resource Support 




Demands 



Resource 

Support 



The website and its management efforts are severely under-resourced. The study 
team believes that ED is facing a critical lack of staff in the area of website creation, 
maintenance, coordination across the various offices, and planning. There are 
insufficient staff in the Office of Educational Research and Improvement (OERI) and 
also in the Principal Operating Components (POC’s) associated with website activities to 
attend to website development and management, and these staff need additional training. 
The current centralized structure has worked effectively for oversight purposes to date, 
but again, this structure is now creating bottlenecks in website content processing, 
evaluation, and ongoing planning. Strategic planning activities have fallen by the 
wayside as staff concentrate on the abundance of day-to-day tasks that must be 
accomplished to keep the website up and running. 

Evaluation efforts have also necessarily remained minimal due to lack of resources. 
The contractor, while successful in maintaining content on the web, appears not to be 
fulfilling contractual obligations regarding submission of appropriate reports and data, 
and has not been evaluated using the performance measures in the contract. Feedback 
from customers (via the website and other customer service activities), which was 
extensively used in early planning and design (and the first redesign) stages has not been 
integrated in a routine way into ongoing site planning activities. The evaluation 
techniques used in this report can provide the department with strategies that should be 
incorporated into ongoing website evaluation efforts. 

As web-based information services and resources grow throughout the Federal 
government, it is likely that more policy attention will be given to the role of Federal 
websites in meeting government- wide information and services dissemination guidelines. 
Although the Department of Education has clearly been working to integrate Federal 
policy into its web policies, it will need to continue to monitor policy developments in 
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this area. Compliance with GPRA, EFOIA, records management guidelines, Y2K, and 
other policy initiatives will likely take on greater importance in the future and will require 
additional staff support. The Department will need to determine how best to assign 
responsibilities to insure agency compliance with these guidelines and requirements. 

Given these concerns, the Department should; 

• Clarify the primary purposes of the website and set priorities for what are the 
most important activities that should be supported via the website. 

• Reassess how website activities are resourced and immediately begin processes to 
increase that allocation. 

• Restructure and examine the responsibilities of key individuals related to the 
website and develop strategies for eliminating potential bottlenecks and pressure 
points (specific recommendations are presented in the management section). 

• Implement a program of ongoing training for staff in basic website skills. 
(Training is required in the application contractor’s contract.) 

• Continue and expand log analyses. More sophisticated analysis software can be 
purchased. Procedures for addressing problems identified in log analysis should 
be implemented. Strategies to encourage webmasters to review and act upon log 
analysis reports should be developed. 

• Develop mechanisms for routinely integrating feedback from customers. 

• Establish responsibilities and procedures to monitor and comply with Federal 
policies that affect website development and use. 

• Develop a policy of ongoing website evaluation. 

The Department will continue to develop its web-based presence. It needs now to 
quickly address problems identified by this report and then to continue to assess how it is 
organized to support ongoing growth and development. In this process, the department 
should consider instituting more formal control and oversight over website development. 



MOVING TOWARDS A USER-CENTERED SITE 

The team’s analysis of logs and user assessment point to a site which has highly- 
used areas, under-used areas, some technical problems which present possible difficulties 
for users, a number of difficulties related to searching and identifying specific 
information, and several design aspects that cater to the expert, knowledgeable user but 
may be less satisfactory to a novice user. This is a not untypical situation for an 
organizational site of this complexity and age. Most organizations first begin a website 
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by recreating the organization’s structure and by providing information in essentially the 
same format as text-based versions of that information. 

To move the Department’s website to the next level of evolution requires that the web 
be perceived as a unique type of information tool that does not need to rnimic 
organizational structure or other information dissemination strategies. By virtue of its 
ubiquity and its enabling of enhanced access for a diverse set of users, a website needs to 
shift in focus away from an organization’s perception of itself to the customers 
perception of it. A user-centered approach to design is necessary. 

A user-centered approach would mean that users are more proactively approached to 
determine their needs (both technical and in terms of information), their perceptions of 
website organization, terminology, ease of navigation, etc., and that site redesign would 
explicitly address those perceptions and needs. Additionally, knowledge about how users 
navigate websites, read online, etc. would be employed. For example, it may be useful to 
have different approaches to the website for those who are beginners versus those who 
are more sophisticated in their web skills and/or knowledge of the Department. It may 
also be important to consider “real time” user support and assistance. These general 
strategies change a site from one which “mimics” print publications to one that is a user- 
centered and user-responsive site (see Section HI for details). 

The Department has already taken a number of steps in this direction. The team s 
specific recommendations throughout the report and the general ones here should enable 
the Department to continue in this direction. To move towards a user-centered website, 
the Department could; 

• Explicitly address and specify the purpose(s) and audiences for the website. 



• Continue to tabulate the results of the online survey and incorporate mechanisms 
whereby the results are used by POCs and others to improve the website. 

• Establish a team (or several teams) of interested users to provide regular feedback 
to the website management on the site. 

• Undertake additional user studies of site terminology (for example, terms on the 
A-Z index could be assessed from a user perspective; one source of data are user 
terms put in on the search engines). 

• Integrate website customer service activities with existing customer service 
activities. 

• Continue log analyses practices and extend using software that produces data such 
as that in Section IV of this report. 



• Develop policies/procedufes for website content additions that incorporate user- 
centered research (e.g., on how people read online; typical user technical 
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configurations) and which lead to a consistent “look and feel” as well as 
consistent content and its organization on the-website. 

• Develop mechanisms beyond the IWG for the organization that would enable it to 
move beyond office-level website initiatives. 

The study team recognizes that shifting a website to a user-centered approach requires 
efforts and resources that go beyond the website itself. This process will not be simple or 
immediate but will require a philosophical shift in the organization that recognizes the 
increasing dominance of electronic and web-based information services and resources as 
the medium of choice. 



STRATEGIES FOR ONGOING EVALUATION 

A purpose of this smdy was to suggest ways in which the Department can 
continue a process for regular ongoing assessment of its website. The study team 
recommends that future assessment efforts continue to incorporate a four-pronged 
approach that incorporates log and transaction analysis, user assessments, management 
assessment, and policy assessment. As shown in this report, the combination of these 
approaches offers a powerful means to provide a comprehensive view for where a 
Federal website can be improved. 

In addition, the study team has provided the Department with detailed assessment 
techniques and data collection instruments. The appendices provide detailed descriptions 
of these methods and data collection instruments. Depending on the needs of die 
Department, the study team believes that many of these techniques and data collection 
instruments can be replicated or modified for future evaluation efforts. At issue is a 
commitment of time and resources to engage in such evaluation efforts. Nonetheless, the 
techniques offered in this report provide a basis for such efforts to be continued and 
refined. 



INCREASING ACCESS TO WEB-BASED RESOURCES AND SERVICES 

Clearly, the Federal government will continue to rely on the dissemination of 
government information and the provision of a range of services via a web and networked 
environment. Indeed, webmasters may not be able, at this time, to even foresee what 
those services might entail. The Department of Education has made significant strides in 
realizing the vision of Vice President A1 Gore as quoted in the introduction to this report. 
That vision, however, requires that Federal agencies rethink how they “do business” and 
“provide services” in this evolving web environment. Part of this “rethinking” is 
allocating adequate resources and obtaining staff with the necessary skills to realize the 
vision. 
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An issue that all Federal agencies must address is the degree to which they can 
simultaneously continue traditional information Services and resources and move to and 
expand their web and networked delivery of information and services. In fact, nmst 
ag^cies are now at the point where it is exceedingly difficult to do more with less. The 
cLent situation at the Department of Education suggests that for at least the web-based 
services and resources, the Department will need to commit additional resources to do 
more. Thus, setting priorities for what traditional services might be dropped or 
redesigned for a web environment will be a key management task in the future. 

Improving the nation’s access to and use of Department of Education information, 
resources and services, can be done via the web environment - indeed, the findings 
presented in this report show that such is clearly the case. The Department, however, 
should expect continued significant growth in the nation’s use of its web as more people 
become familiar with web services, more have computers and high-speed access from the 
home, and as more people expect to be able to conduct business via the web. With sue 
rising levels of user demand and expectations, the Department should build on its existing 
strengths in web-based services and move quickly to respond to the concerns identified in 

this report. 
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I. INTRODUCTION 



Increasingly, Federal agencies rely on web-based technologies to provide a range of 
information resources and services. Indeed, the number and content of U.S. Federal 
websites continues to grow and expand as government agencies develop more uses and 
applications for these websites. The U.S. Department of Education (the Department) is 
one of those agencies that has made a conscious effort to use and develop web-based 
information resources and services as an important means for communicating with 
citizens in this country as well as users around the world. 

The move to a web environment to provide information about the Federal government 
and its activities/programs, to provide direct and interactive services, and to communicate 
directly with users has been encouraged by the Clinton administration through the 
National Partnership for Reinventing Government (formerly the National Performance 
Review) and a range of other initiatives. As Vice President A1 Gore stated in a 1997 
Information Infrastrucmre Task Force Report {Access America, Reengineering through 
Information Technology, Washington DC: Government Printing Office): 

The idea of reengineering through technology is critical. We didn’t want to automate 
the old, worn processes of government. Information technology (FT) was and is the 
great enabler for reinvention. It allows us to rethink, in fundamental ways, how 
people work and how we serve customers... [we will have] a government where all 
Americans have the opportunity to get services electronically and where, aided by 
technology, the productivity of government operations will be soaring. 

Basically the policy message has been for Federal agencies to better exploit the 
networked environment and especially web-based services and resources as a means of 
serving the Nation’s citizens more effectively. 

The Department of Education has been one of a number of cabinet level agencies that 
has, in fact, committed significant effort to developing a website (www.ed.gov) as a 
means of improving access to and use of Department of Education programs and 
services. In light of the growth and development of the Department’s website and related 
web-based services and resources, they contracted with the authors to conduct an 
assessment of the website. The assessment began September 1, 1998 and was completed 
in December, 1998. 



PURPOSE OF THE STUDY 

The goal of the project was to evaluate several of the Department websites in order to 
inform the Department’s on-going efforts to improve these websites. Specific objectives 
that guided the study were: 

• Identify factors . that affect the overall success and usability of selected 
Departmental websites 
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• Examine the processes by which selected websites are managed and coordinated 
across the Department 

• Review the ease with which selected Departmental websites could be navigated 
and the ease with which users could locate and obtain information and services 
from those websites 

. Determine the degree to which the Department has developed an adequate policy 
system for operating the websites and the degree to which these policies recognize 
government-wide policy broadly related to website management and development 

• Provide example evaluation techniques and methods that the Department could 
use/modify for future assessment efforts. 

Underlying all of these objectives was the effort to make constructive suggestions for 
how these websites and the associated management/policy infrastmcture might be 
improved to best assist the Department accomplish agency missions and operate its 
programs effectively. 



STUDY APPROACH 

The investigators believe that to best evaluate a Federal agency website a number of 
different approaches should be used. Thus, the study team employed four approaches to 
assess selected Department websites: 



• Management assessment: This approach begins with a management perspective 
of how the agency is organized to manage Departmental websites in terms of 
developing content, adding and deleting pages, controlling quality, responding to 
customers, and evaluating the website (among other factors). At the Department 
of Education, this assessment was especially important, as there are a number of 
offices, contractors, and individuals who are involved in the organization and 
management of the website. 



• Policy analysis: This approach stresses the de^ee to which the Department has 
developed internal policies to manage the website and the degree to which those 
policies are understandable and implemented. This approach also considers Ae 
degree to which existing agency policies do or do not recogmze government-wide 
policies dealing with such topics as records management, privacy. Freedom of 
Information (FOI), security. Government Performance Results Act (GPRA), etc. 

• Log and transaction analysis: This approach includes a range of assessment 
techniques that rely on data contained in the web server. A number of 
conunercial packages are available or can be developed that will generate 
statistical H^ta regarding the users, error messages, bad links, time on the site, 
links to other sites, etc. In addition, log analysis also includes a number of 
techniques of directly accessing the server and/or its various logs to better 
understand the way in which users use the website. 
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• Usability testing: This approach assesses the usability of the website from the 
perspective of users. Typically users examine the website in terms of content, 
ease of use, navigability, organization, availability and quality of help, etc. 
Usability testing can be done by specific targeted groups where individuals from 
these groups perform searches or attempt to access information from the web and 
provide an assessment of specific aspects of the website. 

Our current view is that a meaningful assessment of a Federal website needs to draw 
upon all four of these approaches as each informs the other. Improved website 
"usability" results from all of the factors in each of the four areas outlined above. 



METHOD AND DATA COLLECTION OVERVIEW 

Members of the study team included the two principal investigamrs, Heit and 
McClure who are members of the faculty at the School of Information Studies; it 
included one adjunct faculty member, Rubin. Two doctoral students, Eschenfelder and 
Taffet, and two masters students. Abend and Pimentel, also participated on the study 
team. Thus, a significant amount of personnel was committed to the study over this four- 

month period. 

The study team developed a time schedule to accomplish study objectives within the 
period of the study. There was some considerable discussion early in the project to 
clarify project goals and activities, agree upon specific websites to assess and speci ic 
Departmental offices for web analysis, and clarify some of the techniques for log 
analysis. There was also an effort to coordinate user data collection efforts at two 
national conference sites. These efforts for user-based assessment at nation 
conferences, however, were not successful. 

Members of the study team participated in two on-site visits to the Department for 
data collection and one visit to provide an executive briefing. They conducted a number 
of telephone interviews and email conversations with key individuals at the Department; 
and they organized a number of sessions with users to conduct usability testing. t 
various points in the study, the investigators employed the following data collection 

techniques: 

• Focus groups 

• Individual/group interviews 

• Examination of Departmental documents 

• Analysis of web server logs and reconstruction of searches 

• User review of websites 

• User surveys and group discussions 

• Policy instrument(s) comparison and analysis 

• Departmental staff survey. 
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Details on specific methods and data collection techniques employed are discussed in the 
appropriate sections later in this report. 

After some discussion between the study team and Departmental officials, it was 
agreed that the study would have a primary focus on the Departmental home website at 
www.gov.ed . It was also agreed that the Office of Elementary and Secondary Education 
(OESE) would receive attention in the tissessment tis a means of demonstrating specific 
assessment techniques that might then be applied to other offices. 

The study team worked closely with a Coordinating Team from the Department that 
included a representative from The Office of the Deputy Secretary, the Office of 
Educational Research and Improvement (OERI), and the Office of the Chief Information 
Officer (OCIO). The representative from OERI also served as the primary point of 
contact for the study team throughout the project. Responsibilities of the Coordinating 
Team included providing the investigators with various project information, reports, and 
policies; assisting them identify and contact potential website users to participate in the 
study; identifying participants for the on-site visits; organizing Department technical 
assistance such as obtaining log files; assisting with logistics for meetings with study 
participants; providing workspace and assistance to the investigators during their on-site 
visits to the Department; and reviewing a draft of the final report. 

The study is limited for a number of retisons. The size and scope of the Department 
website is extensive making it impossible to assess but a small number of the total pages 
available. Thus, the study team focused on specific assessment areas and approaches as 
discussed above and detailed in the following sections. Further, the study encompassed a 
short period of time and had limited resources available for conducting the various 

analyses. 

It should also be pointed out that much of the data collected came from interviews 
and focus group sessions. Such data are self-reported. While the study te^ 
incorporated procedures to check and validate responses as best they could, these self- 
reported data represent respondents’ perceptions of activities. On one hand, these 
perceptions may not be agreed upon by others, or they may not adequately represent 
reality - on the other hand, perceptions often are reality for those working in a particular 
situation. Finally, it should be recognized that members of the study team are not as 
familiar with day-to-day operations and activities as are Departmental staff - thus, some 
findings and recommendations in the report may need to be qualified and put in context 
by those at the Department. 

The study team took steps to ensure the validity of the findings by cross-checking 
responses from individuals against other groups of individuals as appropriate. In 
addition, during interviews and focus groups, an effort was made to elicit responses 
related to earlier findings by offering key issues for participants’ comments, by obtaining 
views from a wide range of stakeholder groups, and by asking for comments on views 
that members of the study team had heard in previous sessions. 
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ORGANIZATION OF THE REPORT 



The study covered a large amount of territory and the resultant report had to be 
organized quickly in a coherent manner. Thus, the study team organized the report by the 
four basic evaluation approaches outlined above. Within each major section, there is 
additional information that describes an overview of findings for that particular 
assessment approach, background information as needed, data collection methods, 
detailed findings, and issues and options. The evaluation of OESE followed a similar 
approach and findings related to OESE are presented within each section. 

The report closes with a brief overview of key themes and issues identified 
throughout the study. In this final section, selected aspects of the findings and 
recommendations are brought together with some broad options for next steps that the 
Department may wish to take. Specific findings, issues, options and recommendations 
related to each of the four assessment approaches, however, will be found in the 
individual report sections. 

A number of appendices follow the body of the report. In addition, two Notebooks 
containing supplemental policy information are provided separately. One is a 
compilation of Department of Education web policies and the second is a compilation of 
selected web policies from selected Federal agencies that may affect the development and 
management of Federal websites. 
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II: MANAGEMENT ASSESSMENT 



Any website evaluation should consider the context in which the website is developed 
and maintained. In particular, an evaluation of management issues associated with the 
website including organizational structure, communication patterns, staffing, etc. is 
essential to the development of recommendations and options that can be implemented by 
the organization. Thus, in this component of the project, the study team collected data to 
answer the following questions: 

• What is the current status of the management activities and management structure 
associated with the website? 

• What do stakeholders in the website perceive to be issues for the ongoing 
maintenance/management of a quality the website? 

The study team’s snapshot of the www.ed.gov website management is that the site 
has grown very successfully since its inception via the mechanisms that have been put in 
place (such as the IWG, the website standards and various policies and procedures, etc.). 
However, the site and its management are reaching a critical point. Resources and 
staffing for website management appear not to have kept pace with the exponentially 
increasing demand for the web content and functionality, nor for associated training and 
planning activities (See Figure II- 1). The Department is still on an exponential growth 
curve in its application of the Internet. A number of Principal Operating Components 
(POC’s) still have a desire to increase their web presence by adding additional content 
and/or using the web as a vehicle for business processes (such as grant submission 
online). This growth can be expected to continue since the ongoing development of the 
web technologies will always provide new opportunities for departmental activities to be 
accomplished via the web. The resource and staffing base has not grown concomitantly. 

Thus, while content and functionality are still being added to the site, generally in 
a timely manner, staff involved in various website management activities feel over- 
burdened, unable to plan strategically, or manage except in crisis mode. Time and 
resources to continue to update skills are also not available. While this pattern may be 
sustainable in the short term, it is likely that problems in the areas identified below will 
grow and impact the ability of the Department of Education to maintain a quality website. 
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Figure II-l Web-Based Service 
Demands VS. Resource Support 




Demands 



Resource 

Support 



OVERVIEW OF KEY FINDINGS 

The study team identified a number of key issue areas related to the Department’s 
management of the website. These are briefly summarized here with further discussion 
in later parts of this section. Options for the Department based on these findings are also 
provided at the conclusion of this section. 

• Purpose of the website: Different stakeholders hold different perceptions of the 
purpose of the website. The website cannot successfully fulfill all these purposes 
simultaneously. 

• Organizational structure associated with the website management: There are a 
wide variety of the website stakeholders throughout the organization as well as at 
least two teams of contractors. A mix of formal and informal management and 
communication structures has developed. These stmctures may not be sufficient 
as the website content and functionality continues to expand. 

• Coordination of communication: Communication is largely “vertical” with each 
POC independently contributing to the website (via the OERI POC liaison). This 
has the potential to lead to: duplication of information on the website, 
inaccuracy/inconsistency of information on the website, and inadequate sharing of 
expertise and insights among POC’::. Communication between agency POC’s 
and the contractor is also “vertical” with the majority of communication flowing 
through the OERI POC liaison or Keith Stubbs and/or his staff and back. 
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Internet Working Group Role: The IWG, while having a charge (as outlined in 
the Nov. 19, 1996 memo from Mike Smith and Leo Komfield), has assumed an 
unclear role in the management of the website. Some IWG members report doing 
much substantive work, others report being unclear what their responsibilities are. 

Contractor Relationships: A number of participants indicated dissatisfaction 
with the Application Contractor’s performance. While the contractor has been 
responsive to a steady stream of requests for quick-turnaround work, other aspects 
of performance specified by the contract have not been fulfilled. The contract 
indicates that the contractor should furnish a wide variety of management reports; 
these have not been provided. Additionally, participants mentioned problems 
with knowing the status of various projects, who was responsible (on the 
contractor’s end), and receiving useful communication via email. Several 
participants pointed out that the contractor should be providing training but has 
not done so. The relationship between the Applications Contractor and the 
Infrastructure contractor may also be a source of coordination problems. 

Website Evaluation: A number of evaluation and approval policies are in place. 
These include the 6 month review by POC’s of their content, the design standards, 
the approval of publications by the Office of Public Affairs and the monthly web 
statistics generated by the contractor and distilled and disseminated by Keith 
Stubbs. However, there is no formalized process for oversight of content 
consistency, accuracy, or clarity (though some occurs on an ad hoc basis). 
Feedback from customer service personnel on user problems and questions 
appears to be underutilized as a source of information for website redesign. 

Website Management Staffing: There are insufficient personnel devoted to the 
website content creation, management, training, and evaluation. Participants, 
almost uniformly, expressed frustration about the number of activities for which 
they and others were responsible. Keith Stubbs has insufficient number of staff 
with insufficient skills. Keith himself has too many job responsibilities, and the 
POC representatives often have the website activities added on to their regular 
duties. 



Management of Customer Relations: The Department has a well-developed set 
of avenues by which the public may comment or ask questions. However, it does 
not appear that information on commonly asked questions (and their answers) is 
well shared across the Department. The website has added another set of avenues 
for customers to ask questions; these can draw on expertise from existing 
customer service expertise as the well provide new models for satisfying customer 
inquiries. 



Strategic Planning for the Website: The Department has an excellent long term 
vision of how it wishes to leverage Internet technology in support of its goals. 
Many staff report, however, that there is little time during their work weeks in 



which to think creatively or strategically. This lack (see staffing above) may 
impact the Department’s ability to continue to grow the Internet site. 



DATA COLLECTION 

The team collected data via interviews and surveys from a variety of stakeholders. 
Data were collected via interviews and surveys. Two members of the study team (Hert 
and Eschenfelder) interviewed personnel at the Department on October 6-7, 1998. The 
objectives were to gather data regarding users of the website as well as to preliminarily 
understand the website’s management structure and the relationships among information 
dissemination activities via the website and other channels at the Department (such as the 
National Library of Education). The study team suggested several types of staff who 
should be interviewed and the Coordinating Team at the Department identified specific 
participants and scheduled interviews. 



During this site visit, the team members worked from interview guides-one that 
focussed on user aspects, including mechanisms for responding to customer inquiries 
(Appendix V-1) and another that focussed on management aspects (exclusive of customer 
service functions) which is presented as Appendix II- 1. For each interview, participants 
introduced themselves and the study team determined which instrument was most 
appropriate given the person’s (or people’s) job responsibilities. The profile of 
participants in presented in Section V of this report. 

These interviews enabled the team to form a preliminary sense of the website 
management. Additional interviews were scheduled in November 1998 (Nov. 4-5). As in 
the previous round of interviews, the study team suggested appropriate personnel and the 
Coordination Team identified specific staff and scheduled the interviews. Study team 
members Hert and McClure jointly and individually interviewed additional agency staff 
with a variety of roles in the website management. These staff included the Internet 
Working Group (IWG) Co-Chairs (3), members of the Internet Working Group (21 in 
large group interview, 5 in small group interview, 2 in individual interviews). 
Applications Contractor Staff (2), Principal Operating Center Liaisons in OERI (5), and 
members of Sally Budd’s staff in OCIO (2). Appendix H-1 is the interview guide. Many 
interviews included a policy component as well and findings from that component are 
presented in Section HI. 



In addition to these on-site interviews, Hert conducted a telephone interview with 
4 members of the Office of Elementary and Secondary Education on December 3, 1998. 
The purpose of the interview was to understand the management of the OESE website 
and how its management (and the website) related to the management of www.ed.gov. 

The study team also prepared a self-administered survey (Appendix II-2) which 
they distributed during the Internet Working Group meeting on November 5, 1998. IWG 
members were encouraged to complete the survey during the meeting and to take a copy 
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to distribute to appropriate staff within their units (to be returned via fax). A total of 24 
surveys were returned in time for analysis. 

Throughout the data collection process, a variety of efforts were made to assess 
the validity of the data collected, including triangulation (of data across multiple 
participants), member-checking (where people are provided with researcher 
reconstructions of the data collected and asked for comment), and debriefing with the 
Coordinating Team at the Department. 



FINDINGS 

Most of the findings presented below represent distillations of information 
gathered via the interviews and survey instruments indicated above. The survey 
considered both management and policy aspects of the site and thus a brief statement of 
findings from the survey is provided separately. The OESE interview is also reported 
separately, though some of its data have also been integrated into the other areas of 
findings. 

Findings from the Survey Instrument 

A total of 24 departmental staff returned a completed survey (Appendix II-2). 
These staff included members of the IWG, various POC personnel and staff with specific 
web management responsibilities. Table II- 1, II-2, and II-3 provide summary information 
from the survey. 

The respondents of the survey bring much knowledge of the Department to their 
web activities, with an average of over 12 years of employment at the Department. 
Involvement in web activities is naturally much lower with an average of 2.5 years. 
When asked about the number of hours spent in various activities (Table 11- 1), 
respondents tended to either spend little time ( 1 -3 hours/week) or a great deal of time 
(more than 8 hours/week) which may reflect different job responsibilities for the 
respondents. Strikingly, when asked about web evaluation activities, almost all 
respondents (N=18) reported that they spent less than one hour per week on this activity. 

Table II-l: Summary of Survey Questions 1-8 
N=24 



Question 




Average of Responses 


1 


Number of years employed by Department of Education 


12.77 


2 


Number of years employed by your current office or 
program 


7.97 


3 


Number of years experience providing information and 
services via the Web 


2.50 


4 


Number of years serving as IWG representative 


1.38 



Questions 5a-5e asked how many hours respondents spent in an average week on five 
activities. The following legend explains the values in the Response column: 

• l=less than one hour 

• 2=1-3 hours 

• 3=3-6 hours 

• 4=6-8 hours 

• 5=over 8 hours 



Question 




Response 


Frequency of 
Response 


5a 


Web planning (e.g. goal setting) 


1 

2 


5 

8 






3 


3 






4 


0 






5 


8 


5b 


Web development (e.g. design, layout) 


1 

2 


8 

7 






3 


2 






4 


1 






5 


6 


5c 


Web management (e.g. maintenance, 


1 


6 




updating) 


2 

3 


4 

3 






4 


3 






5 


7 


5d 


Web evaluation (e.g. customer survey) 


1 


18 






2 


3 






3 


0 






4 


0 






5 


3 


5e 


Other web related activities 


1 


4 






2 


6 






3 


1 






4 


3 






5 


8 



Question 




Average of Responses 


6 


How many people in your office are significantly 
involved in web development activities? 


4.26 
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For questions 7 and 8, the following legend explains the values in the Response column. 



• 1 = 0-100 
• 2 = 101-1000 

• 3=1001-10,000 

• 4=greater than 10,000 



Question 




Response 


Frequency of 
Response 


7 


Please estimate the number of web pages 


1 


15 




which you personally oversee. 


2 

3 


3 

3 






4 


1 


8 


Please estimate the number of web pages 


1 ^ 


11 




which your office oversees. 


2 


3 






3 


4 






4 


2 



Questions 9 through 33 asked respondents to indicate their agreement or 
disagreement with various statements related to web management and web policy. Table 
U-2 summarizes the data by question and table II-3 provides a graphic representation of 
the averages). The average for many questions was neutral (in the middle of the scale) 
(questions 10, 13, 14, 15, 17, 19, 22, 24, 25, 29, 30, 32), only slightly negative (slight 
disagreement with the statement) (questions 11, 16, 18, 21, 28), or only slightly positive 
(questions 20, 26. 27). Neutral answers may mean either that a respondent was not clear 
on how he or she felt or that the status quo was not a problem from his or her perception. 

The negative averages point to a few concerns. A strong negative was reported 
(average = 3.33) for the statement “I currently possess all the technical skills and 
knowledge I need to develop internet based services for my office or program.” This is a 
concern that was also expressed in the interviews. A strong negative was also reported 
(average = 3.48) for the statement “I need a great deal of outside (OERI/contractor) 
support for my office or program’s web management efforts.” An average of 3.09 for the 
statement “The processes by which my office or program evaluated our website are 
effective” was calculated suggesting that evaluation is not well integrated into website 
management at this time for some offices. 

Two statements had averages that were strongly positive. These were The IWG 
should have the following role: A governing body to develop Department of Education 
web policy,” (average = 1.74) and The IWG should have the following role: An advisory 
and educational role,” (average = 1.78). 
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Table II-2: Summary of Questions 9-33 
N=24 



Question 




Average of Responses 
l=Strongly Agree 
5=Strongly Disagree 


9 


I currently possess all the technical skills and knowledge I 
need to develop internet based services for my office or 
program. 


3.33 


10 


I currently possess all the technical skills and knowledge I 
need to manage internet based services for my office or 
program. 


2.75 


11 


I need a great deal of outside (OERI/contractor) support 
for my office or program's web development efforts. 


3.17 


12 


I need a great deal of outside (OERI/contractor) support 
for my office or program’s web management efforts. 


3.48 


13 


I am completely satisfied with the support I receive from 
OERI. 


2.35 


14 


I am completely satisfied with the support I receive from 
our internet applications contractors. 


2.57 


15 


I am satisfied with the quality of my office or program's 
website. 


2.70 


16 


My office or program has a formal management system 
for its web pages. 


2.91 


17 


Procedures for placing information on my office or 
program’s website are 
very clear. 


2.29 


18 


I regularly review web statistical data to better understand 
usage of our office or program website. 


3.09 


19 


My office or program always tests pages for disability and 
low end browser capabilities. 


2.61 


20 


My office or program has established procedures to 
process/answer/forward email inquiries. 


2.18 


21 


My office or program always completes semi-annual 
reviews of its website. 


3.09 


22 


My office or program routinely checks for outdated 
information on its website. 


2.59 


23 


The processes by which my office or program evaluates 
our website are effective. 


3.09 


24 


I am very familiar with the Department of Education’ s 
formal World Wide Web policies and procedures for 
privacy 


2.59 


25 


I am very familiar. with the Department of Education’s 
formal World Wide Web policies and procedures for 
security 


2.57 



O 

ERIC 



27 



13 



Average Response 



26 


I am very familiar with the Department of Education’s 
formal World Wide Web policies and procedures for 
access 


2.14 


27 


I am very familiar with the Department of Education’s 
formal World Wide Web policies and procedures for 
updating content. 


2.22 


28 


The other people in my office or program who are 
involved with web development activities are very 
familiar with the Department of Education’s formal World 
Wide Web policies. 


2.95 


29 


The Department of Education web management policies 
and procedures are effective. 


2.50 


30 


The rWG provides an effective forum to exchange 
information regarding web development at the Dept of 
Education. 


2.36 


31 


The rWG should have the following role: A governing 
body to develop Department of Education web policy. 


1.74 


32 


The rWG should have the following role: An oversight 
authority to approve or block a given office or program’s 
web development projects. 


2.65 


33 


The rWG should have the following role: An advisory 
and educational role. 


1.78 



Table II-3 

Average Responses to Questions 9 through 33 
l=Strongly Agree 
5=Strongly Disagree 



4.00 



3.50 



3.00 



150 



100 



ISO 



1.00 
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Summary of Open-Ended Questions 



Questions 34 through 38 asked the respondents to answer open-ended questions. 
Question 34 asked respondents to identify the single most important change the 
Department of Education could make to its web management procedures. Options 
identified by respondents were to; designate a webmaster, identifying one office or 
individual to lead the project; simplify the web management process; enforce existing 
policies and procedures to a greater extent; and promote communication between staff 
regarding existing procedures and policies. 

In question 35, respondents were asked what is the single most important change 
their program or office could make to its web management procedures. The issue of 
communication surfaced again in these responses: respondents indicated that web 

management procedures need to be disseminated to personnel, and disseminated in 
multiple formats for the widest audience. Respondents also recommended purchasing 
and using website management software and improving the website updating process. 
The need for web planning and coordination to be done by central management in POCs 
was articulated and gave emphasis to the desire for centralized management indicated in 
responses to question 34. 

When asked in question 36 what is the single most important change the Internet 
Working Group (IWG) should make to Department web policy, respondents indicated 
they felt that IWG should be more diligent in monitoring and overseeing the process of 
website management and that it should focus on overall policy and planning. 
Respondents also indicated that IWG should take more of a role in deciding the ultimate 
purpose of the Department's website. In addition, responses indicated the desire for 
business units or offices to be given more of a role in the website development process. 

Respondents overwhelmingly answered question 37, what is the purpose of the 
Department of Education webpage, that the purpose of the Department of Education's 
webpage is to disseminate information — educational information to the Departments 
customers in an understandable, straightforward manner. Some respondents felt that the 
Department's website should act as a gateway to direct users to critical resources, and 
ohers felt the website exists to promote bureaucratic agendas. One respondent indicated 
that the website should build a community of those interested in education information 
and policy. Another wrote that its purpose was to provide faster, value-added services to 
customers. 

Finally, not many respondents chose to answer question 38 which solicited further 
general suggestions. Among the suggestions, though, was one which advocated forming 
a group of external customers to review the Department's website and provide user 
feedback. Other suggestions were to improve the searching capabilities of the website 
and that the Department should create webpages specifically for use by children. The 
opinion that information about web management procedures and policies is not well 
disseminated among staff was again voiced. 
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Findings from the Interviews 
Purpose of the website 

As part of the interviews and survey, the study team asked participants to indicate 
what they thought the purpose of the website should be (or currently was). They offered 
a range of answers that can be summarized into the following categories: 

• Provide access to the Department’s resources and services, to disseminate 
information 

• Serve as a gateway to direct users to critical resources 

• Promote bureaucratic agendas 

• Build a community of those interested in education information and policy 

• Provide faster, value-added services to customers 

The team stimulated discussion in the interviews about these purposes to assess the extent 
to which stakeholders had shared perceptions of the purpose or purposes as well as to 
elicit both negative and positive reactions to various purposes. 

The team found that while most participants did think that the site should provide 
access to resources and services of the Department that there was disagreement over 
whether it was appropriate for the site (or various office sites) to advance particular 
political or social agendas. Some participants felt quite strongly that the website formed a 
very important component of the larger public relationships mission of the Department to 
keep the Department and the state of education in the public eye, while others were 
vehemently opposed to any “politicization” of the content or mission. This issue was 
considered both at the level of the www.ed.gov site as well as the individual office 
website level, with one IWG representative indicating that her office was actively moving 
the website content in the direction of promoting a social/political agenda. Overall, there 
were widely differing views as to the primary goals of the website. 

Organizational structure 

Based on the interviews and reviews of documents, the team developed Figure II- 
2 to represent the basic organizational relationships associated with the management of 
the www.ed.gov website. The graphic represented both parties with responsibilities 
associated with the website content and management as well as communication flows. 

The Co-chairs of the IWG provide oversight of web-related, administrative 
activities. The three co-chairs work closely together and have established a personal and 
warm rapport amongst them. They are assisted in policy development by the Internet 
Working Group which consists of representatives from across the Department and 
includes staff from OERI and OCIO with various the web-related responsibilities. Other 
members of the IWG represent their POC’s on the committee and are also the people who 
work with the POC liaisons (in OERI) in providing content to the web. 
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Figure II-2 Key Positions for InternetAVeb Management 




The actual Webserver is maintained by a contractor (called the Applications 
contractor) funded out of OERI’s budget with Keith Stubbs as the COTR. Other 
components of the Internet infrastructure, (such as telecommunications), are managed by 
a different contractor (the Infrastructure contractor) funded by OCIO. Interview 
participants suggested that the division of responsibilities between these contractors is not 
always clear. 



The Department’s POC’s are responsible for creating and maintaining content on 
the web (except for the highest level pages which is created from content available in the 
ED Initiatives newsletter, with OERI updating the Most Requested Items section, and 
with the IWG responsible for the rest of the content). Each POC has a person (most of 
who are also an IWG representative for the POC), who works with a POC liaison in 
OERI by providing content (which may be marked-up or not). The OERI POC liaison 
provides design suggestions, content suggestions, and other help as requested as well as 
serves as the point of contact between the POC and the applications contractor which 
actually manages the website. POC personnel do not communicate directly to the 
contractor-sending the majority of communication through the OERI POC liaison. The 
OERI POC liaison is responsible for forwarding content to the contractor. 

This structure is somewhat congruent with Departmental structure (as represented 
in the organizational chart located at http://www.ed.g0v/pe0 ple.html#0rg). Several 
differences should be noted. In the Departmental structure, several offices do not report 
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to the Office(s) of the Deputy Secretary or Under Secretary. More importantly, the 
organizational chart does not indicate direct connections among OUS/ODS, OERI, and 
OCIO, the three offices that are most highly visible in the website management structure 
(and in resource allocation). Additionally, responsibilities for various aspects of 
management are not always clear nor is associated oversight. The conclusion of the 
study team is that the current web management structure, to some extent, represents a 
unique partnership and joint leadership style that the three IWG Co-chairs have managed 
to create. 

Participants praised the current leadership of the website for being responsive and 
knowledgeable. They especially praised Keith Stubbs who was often identified as the 
pivotal person in getting the web up and running and keeping it working. Participants 
recognized the hours Keith has committed to the web and some worried that no one else 
had the knowledge to fill-in for him if the need arises. Most participants reported that 
Keith would be the person to turn to if they had problems related to the web activities. 

Our observations of meetings, and data from the interviews also indicate that the 
IWG Chairs work extremely well together and have parlayed good interpersonal 
relationships into a team that brings vision, management expertise, and a wide range of 
constituents together. Through their individual efforts, the website has gone from a 
vision to a well populated and widely used tool. 

Participants raised some concerns about the organizational structure. Along with 
reporting the centrality of Keith’s role in the website management, many participants also 
indicated concern that the organization was overly reliant on his unique expertise. There 
was some concern that too many responsibilities were funneled through his office and at 
times, due to limited staffing, delays and bottlenecks could occur. Each OERI POC 
liaison has an assigned backup however for higher level decision making tasks, these 
backups may not be sufficient. 

There is also little formal structure to support POC’s interaction concerning web 
activities amongst themselves. Individuals have developed informal contacts across the 
Department, including contacts in closely related programmatic areas but there was 
concern on the part of participants that without a more formal mechanism for interacting, 
some opportunities for sharing expertise or information about problems that have arisen 
are missed. 

While not indicated on Figure II-2, many participants, especially those on the 
IWG, also indicated that their job responsibilities had expanded to include web activities. 
According to our participants, most of the POC representatives took on the web-related 
activities in addition to all other activities for which they were responsible. They 
expressed the concern that often web-related activities can not be a first priority due to 
their other work. Some the web-related activities “slip” until they have spare time in 
which to accomplish them. A concern expressed by some was that the individual office 
or POC also had not prioritized the web within the office and thus the solicitation of 
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content, updates, and various other help from other members of the office was often 
difficult. 

Figure II-2 is a map of the management structure associated with the www.ed.go v 
site It is important to consider that a number of additional webservers exist within the 
Department (and external to the Department such as http://pfie.ed.gov (a new site in 
partnership with USA Today, and the Project EASI server) which are not within this 
management framework. The Department has a policy for determining when a server can 

exist separately. 

Coordination of Communication 

As Figure II-2 indicated, communication is largely “vertical” with each POC 
independently contributing to the website (via the OERI POC liaison). Most participants 
reported that information (both the web content and other types of the web-related 
information) flowed from a POC only to OERI, then to the contractor. This has the 
potential to be a bottleneck in the processing of information when the OERI POC 
representatives are inundated with the web-related requests. Additionally, these staff are 
not able to prioritize across the POC requests. When problems arise, Keith Stub^ was 
uniformly mentioned as the person to speak with in order to achieve resolution. This is 
another potential bottleneck depending on Keith’s availability. The POC liaison forwards 
the POC requests to a counterpart on the Applications contractor’s staff. This person then 
distributes the work. This is the third possible bottleneck in the communication process. 

Throughout the interviews, a concern was expressed related to the shanng of 
expertise and insight across the department. Some staff have been identified as 
knowledgeable and are often tapped for their expertise although it is not in their job 
responsibilities to provide Internet training support. These staff are called on in an ad hoc 
(and frequent!) capacity to help others build their web authoring skills, site design skills, 
and resolve day to day problems. Lack of an adequate training budget for web activities 
is a problem. A recent email from Keith Stubbs to the study team (Keith Stubbs to 
Charles R. McClure, 24 November 1998) indicated how small training budgets currently 
are. In Keith’s estimation, his training budget allows him to send a staff member for 
training once on the average of every 8 years. 

The IWG as a venue for information exchange was perceived as being 
inadequate — the meetings are too large or there is insufficient time for the level of 
exchange needed. The verticality of communication has meant that some reinventing o 
the wheel” and “stovepiping” has occurred. Other problems which were mentioned 
included the duplication (and/or inconsistency) of information (one POC representative 
reported that there the were at least 3 different lists (all wrong) of referrals m one area 
because the information was not easily shared). 
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Internet Working Group Role 



Participants expressed different perceptions concerning the purpose and efficacy 
of the IWG. Purposes expressed were as a policy-making group, as a place for learning 
and sharing expertise, and as an oversight committee. 

There was also a diversity of perceptions on the efficacy of the group. Several 
participants reported that the real work of the IWG is done by various IWG taskforces or 
by individuals who are charged with various tasks. Another perception that was 
expressed was that the IWG didn’t accomplish much — in fact it was unclear to several 
participants what their responsibilities were as IWG members. They thought that having 
a “job description” which outlined the responsibilities of an IWG member or a POC 
representative would be helpful in educating them. When asked if the IWG was useful 
for sharing information, there was a sense among the 5 members of the IWG asked this 
question, that there was insufficient time within the meetings for this to be accomplished. 

The survey instrument solicited suggestions about the most important changes the 
IWG could make to web policy. Respondents indicated that the IWG should be more 
diligent in monitoring and overseeing the process of website management and that is 
should focus on overall policy and planning. Responses on the survey indicated that the 
IWG should take more of a role in deciding the purpose of the website. Additionally, the 
IWG should organize or develop more training programs. 

Contractor Relationships 

A number of participants indicated dissatisfaction with the Internet Contractor s 
performance. The contract specifically identifies a wide variety of management reports 
which should be provided on a regular basis. The contractor provides the summary 
statistics from WebTrends but apparently none of the other reports. Additionally, 
respondents who spoke about the contractor reported that it was difficult to deternune the 
status of various requests that had been sent to the contractor, how those requests were 
being prioritized on the contractor’s side and which staff person was responsible for a 
particular request. 

Finally, these respondents expressed dissatisfaction with day-to-day 
communications via email with the contractor staff Oftentimes, incomplete information 
is given in emails making tracking of problems difficult. Respondents indicated that 
these problems made it difficult for them to manage their workflow and to interact with 
other Departmental staff who needed information on the status of particular web 
activities. 

The team was not able to speak with staff from the Infrastructure contractor and 
many respondents did not provide any information on this contractor. Those that did, 
suggested there was poor communication and coordination between these two 
contractors. The mere existence of two contractors with a role in the provision of web 
servers may also be a source of coordination problems. 
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Website Evaluation 



A number of evaluation and approval policies are in place. These include the six 
month review by POC’s of their content, the online customer survey, the design 
standards, the approval of publications by the Office of Public Affairs and the monthly 
web statistics generated by the contractor and distilled and disseminated by Keith Stubbs. 
These provide an excellent start in tracking usage and other concerns with the website. 
The study team identified several other areas for which evaluation is currently ad hoc. 

At this point, the POC’s are responsible for assessing the content on their 
component of the site at least every six months. Between the reviews, some assessment 
occurs on an ad hoc basis. Participants reported that if they notice something is out of 
date or inaccurate, they attempt to address it or send the information to the person who 
they think can resolve it. However, there is no review that stretches across all website 
content. Standards are in place for “look and feel” of content on POC websites, but there 
has been only ad hoc review of how the websites and content of individual websites 
relate to each other, is duplicative, or inconsistent. Some participants expressed a concern 
about having too many “front doors” to the Department, with each POC potentially 
attempting to have its website serve as a front door. Also expressed was the concern that 
“look and feel” differed among the websites. 

Feedback from customer service personnel on user problems and questions also 
appears not to be well integrated into the evaluation process. The team’s interviews with 
customer service personnel revealed a wealth of expertise within the agency, extensive 
sets of referrals, and sets of answers to questions. These customer service personnel have 
a good feel for the particular needs and problems of the customers that they tend to 
service. They have developed strategies to quickly turnaround requests, and to provide 
the best information and referrals. Unfortunately, (while not specifically a web 
management problem), the activities of these personnel are not well coordinated which 
results in duplication of effort, some customers potentially falling through the cracks, and 
misinformation. From a web management perspective, there appears to be no mechanism 
by which the information these personnel have accumulated about customer needs and 
problems can be fed into system redesign. Contractor staff also have a good deal of 
knowledge concerning customer complaints, problems, and needs which has not been 
shared with the department except in an ad hoc fashion. 

Some customer service units, such as 1-800-USA-Leam have a well-developed 
training program and problem tracking system in place. This might form a model for 
similar efforts in the web environment. 

Website Management Staffing 

One of the most clear dimensions of the interviews was the level of frustration 
many participants expressed related to how much effort web content creation and 
management entailed. For most participants, the web was an “add-on” to an already too 



heavy plate of duties and as the desire for more content on the web has increased, this 
work has become burdensome. Several participants did comment that there may be a 
tendency to develop web presences that are not sustainable given currently staffing levels 
and expertise. 

Participants commented fairly uniformly on how well Keith Stubbs and his staff 
perform given the number of responsibilities they have. There is a great ded of support 
for Keith and his staff, and a concern that the Department needs to “clone” Keith and 
provide him with additional staff immediately and additional staff if the web is to 
continue to grow. 

In one group interview, Hert asked participants to indicate the skills necessary to 
do the kind of jobs that the OERI POC liaisons currently do. The skills needed were: 
technical, negotiation, presentation, editorial, layout and design, management, 
organizational (content and time), costing (to tell POC’s how much a given design might 
cost), and political. Knowledge of the department and its customers was also key. In 
other interviews, many participants commented on the lack of technical expertise present 
in the Department. A few people have good skills; they are called on extensively, but 
many POC’s lack personnel with even basic HTML skills, let along the web design skills 
that would enable them to take on more design responsibilities, thus freeing up Keith’s 
staff for other activities. 

Lack of adequate training for web-related tasks was mentioned by several 
participants. The paucity of funds devoted to training was also raised as an issue. 

Strategic Planning for the Website 

The Department has an excellent long term vision of how it wishes to leverage 
Internet technology in support of its goals. Many staff report, however, that there is little 
time during their work week in which to think creatively or strategically. This lack may 
impact the Department’s ability to continue to grow the Internet site. 

Currently, little prioritization of tasks/projects seems to be occurring. Either staff 
need to have less on their plates or a system for prioritization needs to be established and 
disseminated. Clarification of responsibilities for various management activities (such as 
soliciting customer feedback, analysis of web logs, etc.) will assist the Department in its 
ongoing planning and allocation of resources. 

Findings from the OESE Management Interview 

One member of the study team conducted a telephone interview with four 
members of the Office of Elementary and Secondary Education on December 3, 1998. 
Participants in the interview all had an interest in, as well as some job responsibilities 
related to, the OESE website. The interview was structured according to the management 
interview guide in Appendix II- L 
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Participants agreed that currently most of the content of the site, as well as its 
structure, supported those who were looking for funding or who were already grantees. 
The Safe and Drug-free Schools program was thought to attract a more diverse audience. 
There was some discussion of whether the site’s current focus on grantees was 
appropriate, and it was generally considered so, though it was also thought that finding 
ways to extend its reach to other audiences might be worth investigating. 

The relationship of content on the OESE site to the www.ed.goy site was also 
discussed. The participants expressed being overwhelmed by content on the www.ed.go y 
site and were frustrated with the difficulty of finding OESE materials via the 
www.ed.gov homepage. To get to OESE requires navigation through several pages, ^d 
it is also not clear which programs are run out of the OESE prior to that point or by using 
the programs and services button. It was recognized that www.ed.ggy provides a 
valuable front door for users. The desire to have more visibility for news items from the 
office and a greater presence for the office on the homepage was mentioned. 



The discussion also considered management aspects of the OESE website. A 
concern was expressed that within OESE, internet activities or having an internet 
perspective hasn’t yet been integrated into the OESE mainstream. This sometimes means 
that content can not be updated as quickly as desired or that it is not made available in 
forms to go on the web (thus resulting in a delay for markup by the contractor). Staff in 
the office lack technical expertise to do HTML though an effort to create upward 
mobility positions (where people could gain new skills such as these) is under way. 

Currently, the POC representative works with several other staff to get content 
and make it ready for the web. At least one other person in the Office is able to send 
materials directly to the office’s OERI POC liaison. This system seems to be working 
successfully. 

The interviewer asked the participants if the process of going through the OERI 
POC liaison to the contractor was working. One person indicated that it provides a bit of 
a review process which helps to prevent “ugliness” and that so far this process hadn t 
created an undue delay or other difficulties. This person did indicate that it was 
sometimes necessary to “hound” the POC liaison and/or the contractor. 

The relationship of OESE to other POC’s was also discussed. Participants 
indicated that more sharing of information about web activities and problems would be 
desirable. One person said that there should be a group specifically devoted to shanng 
ideas about websites. Other participants indicated that they lacked skills to create 
websites and that communication and training with other POC’s would be helpful. 
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Key Findings 



Overall, the key findings from this interview are that, like www.ed.gov, the OESE 
website activities are under-resourced. This has resulted in less than current information 
on some parts of the site and some offices within OESE not yet having a web presence. 
Internal management of the site, given those constraints, appears not to be problematic. 
There is however, a great need for staff with the appropriate skills and sharing of 
information across POC’s. The dissatisfaction expressed about a presence on the 
www.ed. gov site is also worth noting. 



OPTIONS FOR MANAGEMENT OF WWW.ED.GOV 

Drawing on the data collected via interviews and surveys as well as on the Project 
Team’s background in the website evaluation and management, the team has a picture of 
the www.ed.gov site and its management as being at a critical moment in its 
development. The site has grown quite successfully to date through the efforts of many 
hardworking individuals. As management issues have arisen, a number of structures e.g. 
the IWG) have been put in place to support the ongoing development of the website. The 
continuing exponential growth in the web functionality and demand for a web presence is 
causing these structures to feel strain. Outlined below are a set of options that the 
Department might consider as it looks to improve its web presence. 



Purpose of the Website 

The clarification of purpose of the website for both internal and external 
audiences might facilitate efforts to continue to present an integrated website. Given that 
there are likely to be multiple purposes, it may be helpful to determine which purposes 
take precedence over others as new content and functionality are added. Recent additions 
of servers within POC’s, outside of the www.ed.gov management umbrella also suggests 
that the role of www.ed.gov might need to be reestablished in light of these other efforts. 



Organizational Structure 

The existing structure has several potential bottlenecks-largely in the interface 
area between the Department and the Applications contractor (with OERI and Keith 
Stubbs forming the interface). Building redundancy in that area or decentralizing that 
interface may be necessary as increasing amounts of content needs to flow to the 
contractor and as new functionalities are requested. An assistant to Keith,^ partitioning 
Keith’s responsibilities among multiple people, or enabling individual POC’s to interact 
directly with the Applications contractor might all be employed. 

Such strategies may come at a cost. The current structure, by centralizing 
responsibility and channeling it, allows for a good deal of oversight of content 
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acceptability, etc. If the decentralization strategies indicated above are employed, other 
mechanisms to provide oversight will probably need to be put into place. 

Mechanisms which foster interaction among the POC’s seem warranted based on 
the study findings. These might include monthly training sessions, a listserv for those 
interested in website issues, or presentations by POC’s on their current activities. 

It may also be necessary to consider the establishment of some form of content 
oversight body or staff. Right now, with each POC being able to control its own content, 
there is some likelihood of redundancy or inconsistency of information across the website 
as a whole. In would be worthwhile for someone knowledgeable with all the 
Department’s operations to target several content or programmatic areas that cut across 
organizational boundaries and examine how information related to that area is presented 
on the site. If serious inconsistencies or redundancies are noted, an oversight review 
process might be well worth considering. 



Coordination of communication 

Additional vehicles (beyond IWG meetings) might be put in place to share 
expertise across the agency. The Internet Contractor is supposed to provide training 
services-these could be exploited to enhance internet skills for department staff. 

A system for tracking task status could also be put in place. The TeamSite 
software being considered by the Department may have the necessary functionalities. 
The system should provide a record of when tasks are requested by POC’s, when the 
contractor anticipates their completion, when they are initiated by contractor staff, and 
completed. It should also provide a record of communications related to the task. Such a 
system will also provide useful evaluation information on task turnaround and successful 
completion. 

Customer service information could also be better coordinated. Again, a tracking 
system that could monitor which questions had been answered (e.g.. Frequently asked 
questions) and be shared across the Department would reduce redundant efforts m answer 
the same questions. Placing this information on the web might also reduce questions from 
customers. The information that customer service-oriented staff have acquired about 
customer needs and problems should be formally channeled into evaluation activities. 



Internet Working Group Role 

Members of the IWG seem interested and knowledgeable about the Department, 
and committed to the web. These attitudes are probably being under-exploited given that 
some members of the IWG reported that they were not sure what they are supposed to be 
doing. A “job description” of an IWG representative might be written. This would both 
inform current IWG members as well as enable them to convince their POC’s of the level 
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of effort needed to maintain a viable the web presence. POC’s could use the description 
to better match IWG membership to staff skills. 

Survey results indicated fairly strong interest in fostering the role of the IWG as 
advisory and educator and as developer of policy, while there was not a sense that it 
should be an oversight committee concerning office activities. The IWG is also playing 
the role of developer of policy but it appears that further activities in the area of advising 
and educating might be appropriate. 



Contractor Relationships 

It is critical that the contractor be assessed for contract performance. The contract 
has a set of excellent performance measures with clear rules for their application as well 
as success criteria. The contractor can play a larger role in various training related 
initiatives, offering website development expertise, and providing the data that facilitates 
the Department’s decision-making processes and compliance with GPRA requirements. 
These roles are explicit in the contract; thus this should not present contractual problems. 



Website Evaluation 

The current evaluation efforts of the Department are well worth continuing. Data 
from the customer survey are useful as are the NetTrends report. Other evaluation 
activities might also be considered. Gathering additional information from users, 
certainly the hardest part of any website evaluation, would be helpful. One strategy a 
number of organizations employ is the development of an interested and committed 
group of users who regularly comment on the site and its contents. Finding a mechanism 
for sharing information about customers that access other channels would also be 
appropriate. See Section HI for additional discussion of evaluation. 



Website Management Staffing 

The study team believes that the Department is facing a critical lack of staff in the 
area of website creation and maintenance. This lack runs from personnel with basic 
coding skills to those who can think strategically about the role of the web in the 
organization. Staffing is the most pressing problem that the organization faces with 
regard to the success of its website. Additional staff are needed within the POC s to deal 
with web issues, within OERI to expand the capabilities of that unit, and also at the 
contractor’s end. 
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strategic Planning 

The Department has made a good first effort at strategic planning regarding the 
website. The study team recommends that it continue to follow up on this planning, 
accessing the extent to which that first plan has been realized, developing better 
monitoring/oversight mechanisms, establishing performance measures and quality 
standards for web activities, and continuing the development of such plans and vision 
documents. However, as indicated earlier, staff with the time to do such planning are 
necessary. Priorities may also need to be clarified. 



Relationship of OESE to www.ed.gov 

OESE’s relationship to the www.ed.gov management should be assessed to see if 
it is typical of other offices. Two concerns in the relationship are apparent. Staff in 
OESE could benefit from internet training which would enable them to be more self- 
sufficient in the web activities. This training might come under the auspices of the IWG 
or be provided via knowledgeable staff in various POC’s or through the contractor. The 
other concern is the sense that OESE is not visible on the homepage. The upcoming 
redesign process might look into this concern further as it is one that is probably shared 
among the POC’s. 



CONCLUSIONS 

The www.ed. gov website and its management demonstrate the commitment and 
interest of many individuals in the Department of Education. Unfortunately, these 
dedicated staff are on the verge of being overwhelmed by the amount of work the website 
requires. In this section, the team has highlighted a number of dimensions of that crisis. 
The most critical are the: 

• Lack of adequate staff for various web activities throughout the Department 

• Lack of resources. 

These two key problems work to magnify several other problematic aspects of the 
management of the website including: 

• Lack of agreement on purpose of the website 

• An organizational structure that has several potential bottlenecks 

• Lack of coordination of communication 

• Lack of agreement on the purpose of the IWG 

• The need for additional evaluation efforts and ability to integrate feedback on 
customers and their needs 
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The Department has a very successful website. Department officials will need to 1) 
invest new and significant resources and 2) reconsider the organization and 
responsibilities for key individuals who manage the website if it is to move to the next 
generation of websites. 
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III. POLICY ASSESSMENT 



This section of the assessment had two major goals. The first goal was the 
identification and location of relevant Department of Education web policy documents. 
The second goal was the analysis of the identified documents, which included the 
following tasks: 

• An internal analysis of the contents of the identified Department of Education 
web policy documents for the identification of ambiguities, contradictions, 
duplications, gaps, inconsistencies, enforcement mechanisms and modification 
mechanisms; 

• An implementation analysis of selected Department of Education websites in 
order to determine the degree of compliance with the Department web policies; 

• An external analysis of Department web policy/policy documents in comparison 
to the Federal Web Consortium Guidelines; 

• A preliminary review of Department web policies in the broader Federal 
information policy setting - including a comparison of Department web policies 
to those at selected other agencies, and recognizing and integrating selected 
Federal policies into Department policies. 

The combination of these analyses and reviews offers a broad view of the Department of 
Education’s policy system related to website development and management. Two 
supplemental loose-leaf notebooks compiling (1) Department of Education web policies, 
and (2) web policies from selected other Federal agencies resulted from the policy 
assessment. These two notebooks will be provided to the Department for its review. It is 
important to note that the analyses reflect a particular moment in time for the web 
policies rather than illuminating the decisions that created them. Thus important 
contextual elements that impacted the policy development are not identified. 

For the purpose of this section, the following terms are used. Web Policy is used 
here to refer to the overall web design and internal web management practices of a 
particular federal agency, and may encompass multiple web policy documents. Web 
Policy Document (WPD) is used here to refer to a specific document that is intended to 
govern some aspect of the web design and/or internal web management practices of a 
particular federal agency web site. This is in contrast to a general policy document that 
merely resides on the web. A web policy document may consist of one file or multiple 
files considered to function together as a single document. 



WEB POLICY DOCUMENT IDENTIFICATION 

Overview and Key Findings 

The Department of Education has a fairly well developed set of policy documents. 
There is one primary agency-wide web policy document called U.S. Department of 
Education World . Wide Web Server Policy and Procedures 
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(http;//inet.ed.gov/intemal/wwwstds.html); all other agency-wide web policy documents 
are either implementations of policy (e.g. Disclaimer notices) or are derived from the 
primary WPD. The study team did not find any policies specific to individual 
Department office web sites, but did identify a few policies specific to sites lower in the 
organizational hierarchy (e.g. USNEI, NCES, etc.). Most of those policies are also based 
on the agency-wide policy. 



Method 

In a Federal agency that has created a web policy, there will most likely be a 
single conceptual document (which may consist of more than one file) that is considered 
to be the primary agency-wide web policy document (WPD). The first step in identifying 
and locating federal agency WPD's is to identify and locate this primary WPD. Once this 
primary WPD has been identified and located, then any additional agency-wide WPD s 
should be identified and located. Sub-agency or office WPD's can be identified and 
located by subsequently applying this methodology to that level. In addition to WPD’s, 
general policy documents that may also relate to the web environment should be 
identified and located. 

For this study of the Department of Education website, a knowledgeable staff 
member provided the URL for the primary WPD, informing the study team of both its 
existence and its location. Other methods for identifying and locating the primary WPD 
are also possible. The possible methods include; 

• Via a link from the agency's home page 

• Via information from knowledgeable staff, such as the webmaster 

• Via the agency's search engine 

• Via browsing 

• Via an external web search engine 

• Via a relevant publication 

• Via a link from another web site or another web page 

Once the primary WPD was identified and located, the study team began the process of 
locating additional relevant agency-wide WPD's. The team identified and located most 
of the additional agency-wide WPD's by browsing the Department’s website. The team 
also identified and located a few additional WPD’s by searching the Department s 
website. 

Browsing 

• Look for related documents located nearby (i.e. within the same hierarchy) 

• browse in the hierarchy by using an "Up" button or by truncating the URL (see 
Figure m-l) 



• Move sideways in the hierarchy (i.e. explore the links that are contained one level 
up from the primary WPD or another relevant WPD) 

• Move down in the hierarchy (i.e. explore the links from the primary WPD or 
another relevant WPD to other documents nearby) 

• Look for related documents located elsewhere (i.e. not hierarchically related) 

• Explore the links from the primary WPD or another relevant WPD to other 
documents located elsewhere, as long as they are within the same 
Department/Agency/Office 

Searching 

• Use the agency's own search engine first, followed by an external web search 
engine 

• Search using the URL of the primary WPD as the search term. In a number of 
cases, additional web policy documents will contain a link back to the primary 
WPD. 

• Search next by using applicable search terms, such as: 

- web(site) policy guidelines 
web(site) standards 

- web(site) guidelines 

- web(site) policy/policies 

- disclaimer 

- privacy 



Figure III-l 



Example of URL Truncation 



http://www.ed.gov/intemal/wwwstds.html 

becomes 

http://www.ed.gov/intemal/ 
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For this study, the study team identified the organization-specific and program- 
specific WPD's by searching the Department's website as indicated above. The study 
team identified the general policy documents that relate to the web environment by 
browsing as indicated above. 

After the study team identified the web policy documents and general policy 
documents that relate to the web environment, the list of documents found, along with 
their URL's, was sent to the Department of Education. The Department was requested to 
verify that the list was complete and prioritize the documents on the list (See Figure III- 
2 ). 



The methodology outlined above for web policy document identification is based 
on certain assumptions. First, it assumes that there is a primary agency-wide web policy 
document that exists in writing and is available online somewhere on the agency s 
website. Second, it assumes that similar documents will be located close to each other in 
the hierarchy. Third, it assumes that the indexing method used by the agency s search 
engine allows for retrieval of terms anywhere in the document, i.e. full-text indexing. 
Fourth, it assumes that keyword searching is available via the agency's search engine. 
The search terms provided above might, but will not necessarily, be found as phrases — it 
should be possible to search for the presence of these terms in a document, no matter 
where they occur in relation to each other. 



Findings 

The list of the identified Department of Education web policy documents and general 
policy documents that relate to the web environment is included here as Figure III-2. 
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Figure III-2 

List of Department of Education Web Policy Documents 



A. Agency-wide web-specific policies. 

Title: U.S Department of Education World Wide Web Server Policies and 
Procedures 

URL: http ://ww w.ed. go v/intemal/www stds . html 

Title: Guidelines for Reviewing ED Web Sites 

URL: http://www.ed.gOv/offices/OCIO/intemet/inet7.html 

Title: Review Checklist 

URL: http://www.ed.gOv/offices/OCIO/intemet/inet8.html 

Title: ITIRB Decision on Server Management 
URL: http://www.ed.gOv/offices/OCIO/it/it4.html 

Title; Procedure; Request for New External Web Server/Site 
URL: http;//www.ed.gov/offices/OCIO/intemet/inet3.html 

Title: Establishing Links to External Organizations 
URL; http://www.ed.gov/intemal/xtmlink.html 

Title; Collection Scope and Criteria for Inclusion on Main ED Page of Links 
URL: http://inet.ed.gov/EdRes/edrespol.html 

Title: U.S. Department of Education - Web Site Disclaimer of Endorsement 
URL: http://www.ed.gov/disclaimers/endorse.html 

Title: U.S. Department of Education - Copyright Status Notice 
URL: http://www.ed.gov/disclaimers/copyright.html 

Title; U.S. Department of Education -- Web Site Privacy Security Notice 
URL: http;//www.ed.gov/disclaimers/privacy.html 

Title; U.S. Department of Education - Disclaimer 
URL; http://www.ed.gov/disclaimers/exit.html 

Title; ED-WWW Server and Site Problems: Customer Support Process 
URL: http://inet.ed.gov/intemal/customer.html 
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Figure III-2 (continued) 

List of Department of Education Web Policy Documents 



B. Organization-specific and Program-specific web policies. 

Title: USNEI Web Site Disclaimer and Policies 
URL: http://www.ed.gov/NLEAJSNEI/HPlC.html 

Title: Web Site Disclaimer and Policies - Site Disclaimer 
URL: http://www.ed.gov/NLE/USNEI/HP 1 C 1 .html 

Title: Web Site Disclaimer and Policies - USNEI Site Use Agreement 
URL: http://www.ed.gov/NLE/USNEI/HP 1 C2.html 

Title: Web Site Disclaimer and Policies - USNEI Site Policy 
URL: http://www.ed.gov/NLE/USNEI/HPlC3.html 

Title: National Forum on Education Statistics — Draft Forum Web Site Policies and 
Procedures 

URL: http ://nces .ed. go v/forum/websitepolic ies . html 
Title: Think College — Terms and Conditions 

URL: http://www.ed.gov/offices/OPE/thinkcollege/welcome/terms.html 

Title: Principles of the HEP Web Site 

URL: http://www.ed.gOv/offices/OPE/OHEP/wwwplans.html 



C. General policy documents that relate to the web environment 
Title: Assistive Technology 

URL: http://www.ed.gOv/offices/OCIO/asstech/index.html 

Title: Requirements for Accessible Software Design 
URL: http://gcs.ed.gov/coninfo/clibrary/software.htm 

Title: FOIA/Privacy Act/Information Collections 

URL: http://www.ed.gOv/offices/OCIO/infocoll/info 1 .html 

Title: Information Collection Clearances 

URL: http://www.ed.gOv/offices/OCIO/infocoll/info2.html 
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Figure III-2 (continued) 

List of Department of Education Web Policy Documents 



Title: Guidance on Freedom of Information Act 

URL: http://www.ed.gOv/offices/OCIO/infocoll/info3.html 

Title: Guidance on Privacy Act 

URL: http://www.ed.gOv/offices/OCIO/infocoll/info4.html 
Title: Privacy Act Fact Sheet 

URL: http://www.ed.gOv/offices/OCIO/infocoll/pa.pdf 

Title: Information Retention and Disposal 

URL; http://www.ed.gOv/offices/OCIO/infocoll/info5.html 

Title: Memorandum on Disposition of Departmental Records 
URL: http;//www.ed.gov/offices/OCIO/infocoll/info 15.html 

Title: Information Technology Security Manual 
URL: http://www.ed.gov/offices/OCIO/comps/sec.pdf 

Title; Protecting the Privacy of Student Records 

URL; http://nces.ed.g0v/pubs97/p97527/ABTHEDOC.HTM 



Issues and Options 

When this study was initiated, it was unclear if the Department had a 
comprehensive list of the web policy and general policy documents related to their 
website since no list was provided to the study team. After the Department reviewed the 
initial list of identified documents, the Department verified that the list was complete. 
The study team subsequently found an additional web policy document for the HEP web 
site {Principles of the HEP Web Site, httD://www.ed.gov/offices/ 

OPE/OHEP/wwwplans.html). The fact that the Department verified the initial list as 
complete when it, in fact, was missing a web policy document may indicate that the 
Department of Education does not know the full extent of web policy documents that are 
available on its website. 
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One option would be to (1) create and maintain a complete list of such web policy 
documents and general policy documents that relate to the web environment, and (2) 
organize access to these policies in a more coherent and complete manner. The study 
team committed significant effort into compiling these policy documents. They have been 
printed off the websites and organized into a loose-leaf notebook which is provided to the 
Department as supplemental material. 

Creation and maintenance of a comprehensive list raises another issue as to who 
will be responsible for creating such a list and who will be responsible for maintaining it. 
Additionally, procedures for maintaining such a list would be necessary. For the 
Department of Education, the list found here in Figure ni-2 could be used as a starting 
point. Maintenance of the list could become a part of the regular web page review 
process, and could be worked into the Department’s request form titled "Request to Post 
Files on ED WWW Site." 



ANALYSIS OF WEB POLICY DOCUMENT CONTENTS 

Overview of Key Findings 

One of the key issues affecting policy construction in a general setting, in this 
case the federal government, is that policies are written for an audience which is not 
homogeneous and is in fact composed of several different stakeholder groups. Because 
of the heterogeneity of the stakeholder groups, there are different levels of understanding, 
of procedural matters or technical matters for instance, present in different groups. 
Composing policies which are understandable and salient to all audience members is 

difficult. 

This consideration is not unique to the Department of Education's web policies, 
but it is manifested to a degree here. Writing policy, then, becomes a double-edge sword: 
if policies are written to become understandable to the members of the audience who 
have the least understanding of the policy subject matter and are prone to the least use of 
such policies in their daily work, members of the audience who use the policies on a 
regular basis may find the policies tedious, explaining to too great a degree that which is 
accepted as given knowledge or information. 

Overall, the Department of Education's web policies are well-developed and cover 
a broad range of topics. Clearly, the Department has invested sipiificant energy and 
thought in designing comprehensive web policies aimed at presenting the Department s 
web material in a uniform, logical manner. 
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Given this, and recognizing the challenges inherent to developing coherent 
agency-wide web policy, the following are key findings from this analysis of the 
Department of Education’s web policies: 

• In some instances, the Department's policies use terms that are ambiguous and 
which could be clarified by the addition of further definitions, guidelines, or 
examples. 

• The various job titles, their specific duties, and how each title relates to the other 
may need to be much more specifically and explicitly delineated. 

• There are not many areas in the policies which are inconsistent and even fewer 
which are contradictory. 

• In very few instances in the policies were there explicit directives as to how any 
of the policy tenets would be enforced. The policies lack clear statements as to 
how the policies would be enforced, by whom the policies would be enforced, and 
what sanctions would result from non-compliance. 

• For the most part the policies lacked explicit processes for the collection of user 
feedback, both inside and outside the Department, and lacked explicit processes 
for how the policies could be modified or eliminated based on such feedback. 

These findings will be discussed in detail later and will be followed by the description of 
several options for addressing the issues contained in the findings. 



Method 

The Department of Education policies analyzed in this section were first 
identified by the study team and then reviewed by the Department (see Figure III-2). 
This list of web policies to be analyzed was reviewed by Department of Education 
officials prior to analysis by the study team. In the interest of maximizing available 
resources, the decision was made to limit this analysis to what is a subset of all world 
wide web policies promulgated by the Department. This subset includes the primary 
document U.S. Department of Education World Wide Web Server Policy and Procedures 
(http://www.ed.gov/ intemal/wwwstds.html) as well as further iterations of narrow and 
specific policies developed by the Department and the Offices of the Chief Financial and 
Chief Information Officer (OCF and CIO), including: 

• ED- WWW Server and Site Problems Customer Support Process 
(http://inet.ed.gov/intemal/customer.html) 

• Establishing Links to External Organizations 

(http://www.ed.gov/intemal/xtmlink.html) 

• Collection Scope and Criteria for Inclusion on Main ED Page of Links to Other 
Online Educational Resources (http://www.ed.gov/EdRes/edrespol.htm) 

• Disclaimer of Endorsement (http://www.ed.gov/disclaimers/endorse.html) 

• Copyright Status Notice (http://www.ed.gov/disclaimers/copyright.html) 

• Privacy and Security Notice (http://www.ed.gov/disclaimers/privacy.html) 



• http://inet.ed.gOv/offices/OCIO/intemet/inet 1 .html 

• Guidelines for Reviewing ED Web Sites 
(http://inet.ed.gOv/offices/OCIO/intemet/inet7.html) 

• Review Checklist (http://inet.ed.gov/offices/OCIO/intemet/inet8.html) 

• ED-WWW Server and Site Problems Customer Support Process 
(http://inet.ed.gOv/offices/OCIO/intemet/inet5.html) 

• Procedure: Request for New External Web Server/Site 
(http://inet.ed.gOv/offices/OCIO/intemet/inet3.html) 

• Form: Request for New External Web Server/Site 
(http://inet.ed.gOv/offices/OCIO/intemet/inet4.html) 

• Decision Memo-Centralized Management of Web Servers 
(http://inet.ed.gOv/offices/OCIO/it/it4.html) 

• Exit Icon Disclaimer (http://www.ed.gov/offices/OCIO/exit.html) 

A set of specific criteria for analyzing the policies was developed by the study 
team. This set of criteria provided a framework of analysis with which to examine the 
Department's policies. The criteria included the following topics to be used to analyze 
the policy documents: 

• Ambiguities 

• Contradictions 

• Duplication 

• Gaps 

• Inconsistencies 

• Enforcement 

• Possibility for modification. 

In order to be useful in the analysis, however, these topics first had to be operationalized; 
that is, specific questions had to be developed for each topic so that the analyst examining 
the policies would be able to identify particular aspects of each criteria when examining 
the policies. This operationalization was intended to make the analysis both more 
objective and reliable. Table ni-l contains the set of operationalized policy criteria. 
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TABLE III-l 

Operationalized Policy Criteria 



Ambiguity 


• Can a reasonable outsider infer what the policy is (briefly summarize 
policy)? 

• Can the policy be interpreted in multiple ways (how)? 

• How long is the policy (paragraphs)? 

• Does the policy contain examples to minimize confusion (provide 
quotes)? 

• Does the policy cover one topic or multiple topics 


Contradictions 
(black and white) 


• Do policies appear in the same document which contradict this 
particular policy? 

• Do policies appear in the other policy documents which contradict 
this particular policy? 


Duplication 


• Does the same policy or wording appear more than once within the 
same document? 

• Does the same policy or wording appear in other policy documents? 


Gaps 


• Are there areas where additional guidance and detail is needed? 


Inconsistencies (gray 
areas, not 
contradictory just 
differences) 


• Are different directions given for policies within a document? 

• Are different directions given for this policy in other policy 
documents? 


Enforcement 


• Are there any explicit statements as to how the policy will be 
enforced? 

• Are there explicit statements as to who is to enforce these policies? 

• Are any sanctions for non-compliance made clear? 


Possibility of 
modiHcation 


• Is there an explicit process for collecting user feedback (users 
outside the department) on the policy? 

• Is there an explicit process for collecting user feedback (users inside 
the department) on the policy? 

• Is there a process for modifying the policy? 

• Is there a process for eliminating the policy? 



Using this set of operationalized policy criteria, the analyst examined the list of Department of 
Education policies listed above. This examination included reading the policies multiple times 
for better understanding, applying each operationalized policy criteria to each policy, and 
comparing the prescriptions contained in a particular policy to the others. The findings from this 
process are detailed in Appendix III-l. 



Issues and Options 

Despite the detailed analysis- and suggestions presented in Appendix HI-l, it must be 
reemphasized that the Department of Education has developed a set of well thought-out 



policies governing the ED server websites. The policies cover the spectrum of issues that 
are encountered in facilitating a web presence. A similarly well-developed set of web 
policies for another federal agency was not uncovered during this investigation. 

Still, the Department may want to consider several issues highlighted by this analysis. 
While these issues may be addressed in any number of ways, several options can be 
considered. 

What Are the Guidelines for Archiving or Removing Webpages from De partment 
Servers? 



Sections in the main policy document refer to the need to decide whether webpages 
containing out-of-date information should be removed entirely from the website or 
archived. Responsibility for making such a determination was left up to the parties 
responsible for the particular website by this policy. The issue of dealing with old 
webpages is an important one and should not be left up to each office or program to 
decide. The management of out-of-date information should be centralized and not 
allowed to fall where it will. The Department may want to consider several options in 
addressing this issue: 

♦ Creating a set of specific guidelines to be used in determining what material 
should be archived and what material should be removed. 

♦ Designating a centralized place to store all material needing to be archived. 

♦ Designating a job title (or adding specific duties to an existing job title) to oversee 
the management of archived material. 

An explicit policy on this issue would seem to better suit the overall purposes of the 
Department since it would add another centralized policy to the Department's growing 
arsenal. 

Can the Department Create Better Policies and Buv-In From Emp loyees by Facilitating 
User Feedback? 



In the main policy document, U.S. Department of Education World Wide Web Server 
Policy and Procedures, no explicit direction is given as to how the Department may go 
about soliciting and collecting feedback from: users of the Department's websites; and 
Departmental users of the web server policies. Creating specific processes for collecting 
and using user feedback may contribute positively to the future development of the 
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overall web policies and websites. The Department may want to consider several options 
in addressing this issue: 

• Creating a specific policy within the main policy document that prescribes how 
and when external user input should be collected. 

• Creating a specific policy within the main policy document that prescribes a 
process for implementing user input. 

Soliciting and utilizing internal user feedback regarding web policies could have the 
positive effect of creating internal user buy-in, facilitating not only more complete 
implementation and adherence to the web policies, but perhaps better future web policies. 
The same positive effects may also be gained by soliciting the input of external users. 

From Where does the Authority for Education Office Web Po licies Originate? 

In several instances during this analysis, authority for policies promulgated by 
Department offices could not be directly identified in the main policy document, U.S. 
Department of Education World Wide Web Server Policy and Procedures. This situation 
does not readily support a Departmental goal of centralized policy. Policies promulgated 
by offices must find their authority from direction in the main document. The 
Department may want to consider: 

• Reviewing existing Department office policies and identifying the specific places 
in the main policy document which authorize the development of such policies. 

• Eliminating office policies that do not spring from the main policy document. 

• Creating new directives within the main policy document to support office 
policies that are currently without direct authority. 

If offices continue to develop policies that lack a basis for authority in the main policy 
document, the main document's strength will be undermined and the Department may 
find its web policies devolve toward decentralization. 

Who is Responsible for Making Changes and Coordinating the O ngoing Review of the 
Department Websites? 

While subsequent policy documents. Guidelines for Reviewing ED Web Sites and 
Review Checklist, address this issue, the main policy document, U.S. Department of 
Education World Wide Web Server Policy and Procedures, contains no similar explicit 
statements of responsibility. This issue really sums up the lack of enforcement evidenced 
throughout the main document. The Department may want to consider promulgating 
additional policies to address this issue. 
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Such a policy might include clear statements as to; 

• Who exactly is responsible for reviewing Department websites. 

• How websites will be reviewed to ensure that they comply with the stated policies 
of the main policy document. 

• How often such a review should occur. 

• Who will be responsible for implementing recommended changes. 

• Who will be responsible for following up to ensure that recommended changes 
were actually implemented. 

• WTiat are the sanctions for not implementing changes. 

The tasks of implementing recommended changes to websites and following up to ensure 
that such implementation occurs is important. If changes are not implemented, policies 
may then exist with no enforcement and be rendered impotent. The Department needs to 
devise a mechanism to ensure that reviews that lead to implementation of proposed 
changes in order to conform to policy are occurring. 

Has the Department Webmaster and the Webmaster Duties Been Des ignated as Clearly 
as Possible? 

There was evidence throughout the main policy document, US. Department of 
Education World Wide Web Server Policy and Procedures, that inconsistencies exist 
regarding the webmaster duties being shared by OCIO and OERI. Inconsistencies on 
paper often translate into confusion in action. It seems important to identify precisely 
who is ultimately responsible for overseeing the web development and management for 
the Department. The Department Webmaster should function to encourage unity among 
the many disparate websites administered by the Department. It seems unlikely that the 
most effective formulation of this role is created when the role is divided between two or 
more Department offices. The Department may want to consider several options 
regarding this issue; 

• Eliminating inconsistencies in references within the main policy document to 
Department Webmaster duties. 

• More clearly defining the exact duties of the Department Webmaster. 

• Designating the Department Webmaster role within one Department office. 

WTiile the last option is the most dramatic and will require altering existing policy, the 
Department may find that positive results would accrue from taking this step. A clear 
centralization of authority may make possible better enforcement of web policies. 

What are the Exact Duties of the Various Job Titles and How Doe s Each Relate to the 
Others? 

There are a number of job titles that are directly affected by the Department's web 
policies. These include; Department Webmaster, Principal Office Assistant Secretary, 
POC Home Page Editor, and Web Publisher. Importantly, the duties listed for each of 
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these job titles given in the main policy document are not complete lists and they are not 
very specific. Neither do the job title descriptions provide an understanding of how each 
job title really relates to the others in the sense of an organizational hierarchy. The 
Department may find that it is better able to implement its web policies by clarifying the 
duties and relationships contained in these job titles. Along this line of consideration, the 
Department may want to consider several options in addressing this issue: 

• Precisely defining the job duties of each job title involved in web management 
and web development. 

• Creating a detailed organizational diagram of the hierarchy of each job title to the 

others. • i j 

• Making explicit the relationship of each job title to the others. This would include 

delineating who is responsible to whom or defining which job title oversees the 
activities of other job titles. 

• Including all of the above information in the main policy document. 

Making more explicit these relationships should allow existing policies to be better 
implemented and enforced. Continued ambiguity on this issue will probably hold the 
Department back from implementing to the fullest extent its well thought-out policies. 



IMPLEMENTATION ANALYSIS 

Overview and Key Findings 

Overall, most policies appear to be implemented successfully on the websites 
reviewed. Plain text equivalents to graphics are clearly evident throughout the entire site. 
The use of frames is rare on the sites reviewed, but where it does occur, text alternatives 
are also present (http://www.ed.gov/offices/OESE/SIP/programs/). The use of online 
forms is also rare on the sites reviewed, but alternatives are available where forms exist 

(http://www.ed.gov/comments/problemform/ProblemForm.html). 

Additionally, the Department of Education is to be commended for the 
tremendous degree of cross-linking that is evident between web sites for different offices, 
organizations and programs. Responsibilities that are shared between offices result in 
cross-linking to existing program pages or documents rather than duplication. Cross- 
links of this kind are frequently seen in relation to the budget: 

• The Office of Elementary and Secondary Education, and its component 
Compensatory Education Programs are responsible for administering Title I 
programs under the Improving America's Schools Act (LAS A), but the Budget 
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Service of the Office of the Under Secretary is responsible for managing the 
actual funds involved. 






http://www.ed.gov/offices/OESE/program.html links to 
http://www.ed.gov/offices/OUS/title 1 .html 

http://www.ed.gov/offices/OESE/CEP/info.html links to 
http://www.ed.gov/offices/OUS/us98.htm 

Additionally, if a relevant document already exists elsewhere on the ED web site, it will 
be linked to rather than duplicated: 

• Budget News at http://www.ed.gov/offices/OUS/budnews.html links to 
Education Department Testimony before Congress on the FY 1 999 Budget at 
http://www.ed.gov/Speeches/99budtst.html. 

The Department of Education has done an excellent job of leveraging the utility 
of the information resources they have placed online. 

The study team did identify several problems: 

• The cross-linking described above can cause navigational problems due to the 
presence of icons or text links that don't take you back to where you came 
from. Instead these icons take you to the logical home page for that cross- 
linked program, document or collection, and/or the ED home page. 
Experienced users might take note of this and are therefore less likely to get 
lost, but the average user could easily get lost among these cross-links. 

The document Education Department Testimony before Congress on the 
FY 1999 Budget at http://www.ed.gov/Speeches/99budtst.html mentioned 
above links back to the main Department web site 

(http://www.ed.gov/index.html) and to the Speeches and Testimony 
directory kept there (http://www.ed.gov/Speeches/index.html), but does 
not include a link back to Budget News or any other page on the OUS site. 

- The OLCA Home Page (http://www.ed.gov/offices/OLCA/) includes an 
icon called "Budget" which links to The Department of Education Budget 
at http://www.ed.gov/offices/OUS/budget.html. The Department of 
Education Budget page in turn includes a link to OUS, but no link back to 
OLCA. 

- The OLCA Home Page (http://www.ed.gov/offices/OLCA/) also includes 
an icon called "Grants" which links to The Grants and Contracts/Grants 
Redesign at http://www.ed.gov/offices/OCFO/gcinfo.html. The Grants 
and Contracts/Grants Redesign page in turn includes a link to the main 
Department web site, but no link back to OLCA. 

- The OLCA Home Page (http://www.ed.gov/offices/OLCA/) includes 
another icon called "Student Aid" which links to the OPE Home Page at 



http://www.ed.gov/offices/OPE/. The OPE home page does not include a 
link back to OLCA. 

US Department of Education FY 1999 Budget 
(http://www.ed.gov/offices/OUS/Budget99/) includes a link to President s 
State of the Union Address and Education Initiatives at 
http://www.ed.gov/updates/inits98/. The document President's State of the 
Union Address and Education Initiatives includes a link back to the main 
Department home page, but does not have a link back to any page on the 
OUS site. 

• The vast majority of web pages are longer than 1.5 screens, and many are 
longer than 10 screens without being broken into multiple files; documents 
that are broken into more than ten multiple files often do not include a 
downloadable PDF, Microsoft Word or ASCH text version. Examples include: 

Office of Elementary and Secondary Education: Programs & Funding 
Opportunities at http://www.ed.gov/offices/OESE/program.html 
U.S. Department of Education/Office for Civil Rights: Publications And 
Products at http://www.ed.gov/offices/OCR/ocrprod.html 
The Federal Budget Process and Its Implementation in the Department of 
Education at http://www.ed.gov/offices/OUS/budpro2.html 
- GUIDANCE ON STANDARDS ASSESSMENTS, AND 
ACCOUNTABILITY at 

http://www.ed.gov/offices/OESE/StandardsAssessment/ has no 

downloadable version 

Collective Bargaining Agreement at 

http://www.ed.gov/offices/OM/toc.html, contains 43 files but no 
downloadable version 

• Links leading down in the hierarchy are generally fine, but there are a number 
of problems with links leading up in the hierarchy (in addition to the cross- 
linking problem noted above), such that links leading upward are either 
missing or made at the wrong level in the hierarchy: 

a) There is a general tendency among the sites reviewed for key personnel 
biographical pages to be linked only to the ED home page rather than to 
the respective office home pages. Examples include: 

Norma V. Cantu at http://www.ed.gov/offices/OCR/cantu.html 
Scott S. Fleming at http://www.ed.gov/offices/OLCA/fleming.html 
Marshall S. Smith at http://www.ed.gov/offices/OUS/smith.html 

b) There is a tendency for online documents that appear to have counterparts 
on paper to not be linked back to the ED home page or any other page on 
the ED web site; included here are most noticeably legal documents, 
memorandums and letters and other such documents. Examples include: 
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Two letters from the Office for Civil Rights dated July 23, 1998 at 

http://www.ed.gov/offices/OCR/bowlgm.html 

OM: Complaint and Motion at 

http://www.ed.gov/offices/OM/complain.html and 
http://www.ed.gov/offices/OM/motion.html 

- Letter from Leroy Rooker dated November 25, 1997 at 
http://www.ed.gov/offices/OM/hope.html 

- Letter from Scott S. Fleming dated May 8, 1998 at 

http://www.ed.gov/offices/OLCA/eoz2.html 

21st Century Community Learning Center- fact sheet at 

http://www.ed.gOv/offices/OLCA/2 1 stfacts.htm 

c) There is a general lack of compliance in older documents, especially the 
ones that appear to be nothing but ASCII text turned straight into HTML. 
Examples include (all from the Office of Civil Review): 

[/.S. Department of Education/Office for Civil Rights: Letter of 
Findings to University of California, Berkeley School of Law 
(09/25/92 ) and settlement agreement Subject: Considerations for 
determining permissibility of affirmative action admissions program 
under Title VI of the Civil Rights Act of 1964. File Update: 

09/22/97di http://www.ed.gov/offices/OCR/ucberkle.html 
U.S. Department of Education/Office for Civil Rights: Title VI of the 
Civil Rights Act of 1964; Final policy on race-targeted aid (02/23/94). 
Non-discrimination in Federally assisted programs. Replaces the 
proposed policy guidance dated December 10, 1 992 and published in 
the Federal Register, at http://www.ed.gov/offices/OCR/racetarg.html 
U.S. Department of Education/Office for Civil Rights: Letter to 
College and University Counsel confirming policy guidance on race- 
targeted financial aid in light of Podberesky and Adarand decisions. 
(09/07/95). Title VI of the Civil Rights Act of 1964 at 
http://www.ed.gov/offices/OCR/podberes.html 
- U.S. Department of Education/Office for Civil Rights: Memorandum 

to UCLA. Subject: Title VI of the Civil Rights Act of 1964: 
Investigation of the UCLA undergraduate admissions program 
regarding discrimination against Asian American applicants to the 
College of Letters and Science (L & S) and the School of Engineering 
and Applied Science (SEAS), at 
http://www.ed.gov/offices/OCR/ucla.html 

U.S. Department of Education/Office for Civil Rights: Memorandum 
to College and University Counsel (6/30/96). Subject: Title VI of the 
Civil Rights Act of 1964: Consideration of race in making college and 
university admissions decisions and granting financial aid. File 
Update: 07/22/97 at http://www.ed.gov/offices/OCR/dearcol.html 
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U.S. Department of Education/Offtce for Civil Rights: Resolution 
Letter to the Washington Legal Foundation (02/21/97), and Florida 
Atlantic University Resolution Agreement . Subject: Race-based 
scholarship and compliance with the Title VI of the Civil Rights Act of 
1964. File Update: 09/22/97 at 
http://www.ed.gov/offices/OCR/fau.html 
- U.S. Department of Education/Office of the Secretary: Dear Colleague 
Letter from the Secretary of Education Riley (03/19/97). Subject: 
California Civil Rights initiative, Proposition 209, regarding the 
responsibilities of California's school districts and colleges under 
educational programs and civil rights laws administered by this 
Department. Title VI of the Civil Rights Act of 1 964. Affirmative 
Action. File Update: 09/23/97 at 

http://www.ed.gOv/offices/OCR/prop209.html 

U.S. Department of Education/Office for Civil Rights: Memorandum 
to Honorable Rodney Ellis, Texas Senate (4/11/97). Subject: Title VI 
of the Civil Rights Act of 1964: Hopwood v. Texas at 

http://www.ed.gov/offices/OCR/hopwood.html 
U.S. Department of Education/Office for Civil Rights: Memorandum 
to the Oregon State System of Higher Education (07/03/97), and 
Resolution Agreement. Subject: Tuition Waver Program compliance 
with the Title VI of the Civil Rights Act of 1964. File Update: 

09/22/97 at http://www.ed.gov/offices/OCR/oregon.html 
- 1998 UNITED STATES DEPARTMENT OF EDUCATION U.S. 

ACCREDITED POSTSECONDARY MINORITY INSTITUTIONS (l)at 
http://www.ed.gov/offices/OCR/minlistl998.html 

• There is an inconsistency in compliance with the guideline that prescribes a 
link to the Adobe Acrobat Reader home page where PDF files are found. 

- Pages that comply include: 

■ Education Legislation at 

http://www.ed.gov/offices/OLCA/legislation.html 

■ Family Policy Compliance Office at 
http://www.ed.gov/offices/OM/fpco.html 

■ Weekly Job Vacancies at http://www.ed.gov/offices/OM/jobweb.html 

- Pages that don’t comply include: 

■ Comments requested on Draft Consolidated P erformance Report 
Under 0MB Review at 

http://www.ed.gov/offices/OESE/webpara2.html 

■ OLCA Home Page at http://www.ed.gov/offices/OLCA/ 

■ Financial Aid Evaluation at 
http://www.ed.gov/offices/OUS/eval/finaid.html 
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■ Continuous Improvement Management Guide for 21st Century Schools 
at http ://w w w .ed. gov/offices/OU S/e val/2 1 cent/2 1 stc . html 

■ Goals 2000 Legislation and Related Items at http://www.ed.gov/G2K/ 

• There is a general lack of compliance when it comes to external links and 
disclaimers. This may be due in part to the inconsistencies and contradictions 
relating to external links and disclaimers that are found in the primary WPD. 
Pages that include primarily external links but do not have disclaimers or 
explicit notices of the external links include: 

Office of Elementary and Secondary Education: Technical Assistance — 
Hot Topics! at http://www.ed.gov/offices/OESE/hot.html 
U.S. Department of Education/Office for Civil Rights: Related Links at 
http://www.ed.gov/offices/OCR/links.html (this page includes a statement 
which could be viewed as a short disclaimer, but the word “Disclaimer’ is 
not found anywhere). 

• The online version of U.S. Department of Education World Wide Web Server 
Policy and Procedures available at http://inet.ed.gov/intemal/wwwstds.html 
does not match the Microsoft Word, Adobe Acrobat or ASCII versions of the 
same document available for download at the same URL. The online version 
of the document, on the References page of the Technical Standards and 
Guidelines (http://inet.ed.gov/intemal/wwwstds/references.html) includes no 
mention of the Bandwidth Conservation Society under the links shown for 
Design. The downloadable versions all include this reference to the 
Bandwidth Conservation Society. The online version and downloadable 
versions of the same document differ, but there is no way to determine which 
is more recent between them. 

Appendix ni-4 details findings and suggestions resulting from the implementation 
analysis. 



Method 

The web policy created by the Department of Education contains two major types 
of policies — those that govern internal web management practices, and those that govern 
web design principles. The policies that govern internal web management practices 
typically are not reflected visibly in the web pages themselves, so the Implementation 
Analysis was limited to policies that govern web design principles. 

In preparation for the Implementation Analysis, the team prepared an 
Implementation Review Checklist (See Appendix III-2). During a review of the prim^ 
WPD, statements that presented clear prescriptive directives were marked. Prescriptive 
statements tend to contain certain directive words such as must (not), need (to)Ineeds (to). 
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required, shall (not)lshould (not), have to! has to, will, avoid, make sure to, etc. Not all 
such prescriptive statements were chosen. 

Statements that were not chosen included a) statements that were inconsistent 
with other statements, b) statements that would require viewing the HTML code itself m 
order to verify compliance, c) statements that would require subjective judgments on the 
part of the analyst, d) statements where it was possible to verify compliance but was not 
possible to verify non-compliance (e.g. "When updating a page, make sure to update the 
last update date and initials of the individual performing the update which appear at the 
bottom of many pages.") and e) statements that were judged to require too much time for 
review (e.g. "no obvious spelling errors" — the Doctor HTML utility found at 
http://www2.imagiware.com/RxHTML/ is able to check for spelling errors among other 
problems, but is limited to checking four pages per online session, so usage of this utility 
w£is dropped). 

After the selection of prescriptive statements was complete, many were combined 
and summarized in order to simplify the list of statements. For example, rather than 
specifying which format should be present for online viewing, for offline display or print, 
or for offline analysis and manipulation, a single summarizing statement was created — 
"File format matches the intended usage (used online, offline print or display, offline 
analysis or manipulation)." This is a summary statement suitable for reviewing 
purposes, but the detail is found elsewhere. 

Some statements were changed from a form suitable for prescription to a form 
suitable for review (e.g. "Pages whose focus is linking to external sites shall include or 
link to the text of the standard Disclaimer of Endorsement statement ... and a statement 
of the scope of the collection of external links and the criteria for inclusion." becomes 
"Pages whose focus is linking to external sites include or link to 1) the text of the 
standard Disclaimer of Endorsement statement and 2) a statement of the scope of the 
collection of external links and the criteria for inclusion."). The simplified list of 
statements was compiled and statements were logically grouped together. The 
Implementation Review Checklist is attached as Appendix III-2. 

The process of preparing the Implementation Review Checklist took considerable 
time because the Department's web policy is quite well developed, with many documents 
to review. This review and the review checklist are beised solely on the U.S. Department 
of Education World Wide Web Server Policy and Procedures because this primary WPD 
was found to be the basis for or the source for the majority of the remaining web policy 
documents. 

Once the Implementation Review Checklist was complete, five Department of 
Education offices were selected for review. The selection was based on the 
organizational chart found at http://www.ed.g 0 v/pe 0 ple.html# 0 rg. A brief review of die 
organizational chart showed that there are five major organizational areas, including 
External Relations, Budget, Policy & Planning, Decision/Strategy Support, Operations, 
and Programs. The analyst initially chose to review one office from each organizational 
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area, but this was problematic as no office web site was found for Decision/Strategy 
Support. Therefore, the team then chose to review two office web sites from the 
Programs area as this organizational area is an important one. 

Offices chosen included the Office of Legislative and Congressional Affairs (to 
represent External Relations), the Office of the Under Secretary (to represent Budget, 
Policy & Planning), the Office of Management (to represent Operations), the Office of 
Elementary and Secondary Education and the Office for Civil Rights (both representing 
Programs). As the Office of Elementary and Secondary Education is mentioned 
elsewhere in this study, this office was a mandatory review rather than a choice. Pages 
chosen for review within each office were for the most part limited to those pages that 
were obviously hosted by that site as opposed to coming from some other site within ED 
or from other government sites (i.e. they shared the initial portion of the URL in 
common, such as http://www.ed.gov/offices/OESE). Exceptions to this general rule were 
made in order to review the home pages for sites at the Organization/Program level, and 
to explore certain cross-links. 

In the initial stages of reviewing the various sites chosen, each analyst filled out a 
grid based on the Implementation Review Checklist for each web page viewed. A 
sample grid is attached as Appendix III-3. As the process of filling out these grids 
became very time consuming, the team soon began to review more quickly by making a 
bookmark for each page viewed and making notes of problems relating to the 
Implementation Review Checklist items that were encountered during the review. 

The methodology outlined above for Implementation Analysis might vary when 
applied to other federal agencies because adjustments might be necessary if the web 
policy is either less developed or more developed than the web policy found at the 
Department of Education. The checklists developed for this analysis may be revised and 
updated by the Department for future use. Appendix III-4 presents details of the findings 
from the implementation analysis. 



Issues and Options 

The tremendous degree of cross-linking present on the Department's website is 
both a help and a hindrance when it comes to navigating back and forth between offices, 
organizations and programs. Options for solving this problem are as much philosophical 
and policy-driven as they are technical. 

There are clearly a number of older documents on the Department's website that 
predate the Department's current web policy; many of these older documents do not 
comply with the current policy. The available web policy documents do not address the 
timing or procedure for getting these older web pages into compliance with the current 
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policy. The Department needs to address this situation. Options for addressing 
this issue include: 

• Providing a grandfather exclusion to previously existing pages so that they don t 
have to comply. 

• Creating and publishing a timetable and procedure for getting the older pages into 
compliance — this could perhaps be worked into the periodic reviews of the web 
site 

Links that do or should go up in the hierarchy are problematic for a number of 
different reasons as explained previously in this section. Options for addressing these 
link problems include: 

• Addressing the issue of older pages that predate the policy as stated above. 

• Following the very good example set by the biographical page for Patricia W. 
McNeil, Assistant Secretary for Vocational and Adult Education 
(http://www.ed.gOv/offices/OVAE/mcneil.html). This page links to both the main 
ED home page and the individual office home page. 

• Incorporating this into the periodic review of web sites 

The ED guidelines aren't consistently followed in terms of providing a link to the 
Adobe Acrobat reader where PDF files are found. Possible options for addressing this 
problem include: 

• Adding a section about Adobe Acrobat to the Standard Footer so that the Adobe 
Acrobat reader will be linked by default. 

• Following the elegant solution demonstrated by the U.S. Patent and Trademark 
Office, which has created a page called Document Formats and Viewers 
(http://www.uspto.gov/web/menu/viewers.html). This single page includes 
information on all proprietary document formats and links to readers where 
available. Should ED decide to follow the U.S. Patent and Trademark Office 
example, care must be taken to ensure that the page created is available as a link 
either from all web pages or from all web pages that include downloadable 
documents in proprietary formats. 

The guidelines with regard to disclaimers are clearly not followed most of the 
time, perhaps due to the inconsistencies and contradictions present in the web policy. 
Options for addressing this situation include: 

• Addressing the inconsistencies and contradictions found in the web policy - 
with clear guidelines to follow, more web pages should be in compliance. 
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• Following the very good example set by the Office of Hearings and Appeals 
(http://www.ed-oha.org/), a component of the Office of Management. The 
Office of Hearings and Appeals presents a Disclaimer link right from the 
home page. 

The team found one instance where the online and downloadable versions of the 
same document are not identical. It is possible that there are other documents that also 
have this same problem. The Department’s policy does not include a procedure for 
keeping online and downloadable versions of the same document in sync with each other. 
A procedure for keeping online and downloadable versions of the same document in 
synch with each other must be developed. One option might be; 

• Include a section on the form Request to Post Files on the ED-WWW Site that 
could be used to indicate if a downloadable version is present and whether 
changes to it are needed. 

These and other proposed options and suggestions should be considered as a means to 
simplify and better coordinate website policies as they affect various units across the 
Department. 



COMPARISON OF DEPARTMENT OF EDUCATION POLICY 
GUIDELINES TO FEDERAL WEB CONSORTIUM GUIDELINES 



This section compares the Department of Education web policies. World Wide Web 
Server and Procedures (www.ed.gov/intemal/wwwstds.html) to The Federal Web 
Consortium guidelines (http://www.dtic.mil/staff/cthomps/guidelines/) . The Consortium 
guidelines were originally based on the primary ED WPD (see article at 
http://www.ed.gov/pubs/OpenWindow/1997Fall/staff.html). Since they were first 
created, the Federal Web Consortium guidelines have undergone one revision, and a 
second revision is currently underway. This comparison is based on the draft version of 
the second revision. 



Overview and Key Findings 

The side-by-side analysis between Consortium guidelines and Department 
policies identified the following key findings: 

• The Federal Web Consortium feels that web policy creation should involve 
more personnel than just the webmaster. 

• The Federal Web Consortium Guidelines include a very useful checklist, 
inclusion of a similar checklist would add significantly to the utility of the 
Department’s web policy. The checklist created by the Department’s Internet 
Working Group is a step in the right direction. 
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The Federal Web Consortium Guidelines show a greater sensitivity to users 
when they allow personal e-mail addresses to be used for contact where 
personal service is rendered. 

The Federal Web Consortium Guidelines state that Individual Home Pages are 
allowable if they meet certain criteria. While the Department’ s policy strictly 
prohibits Personal Home Pages, the Department’s web site does include short 
biographical pages for key personnel. In this sense, Individual Home Pages 
aren’t entirely forbidden by the Department. 

The Federal Web Consortium Guidelines provide additional information 
specifying in what manner external links can be made explicit. The 
Department’s policy states that external links should be clearly identified, but 
does not indicate what form such identification should take. 

Both policies address Restricted Access, but the Federal Web Consortium 
Guidelines go much further in explaining when restricted access might be 
called for. The Federal Web Consortium Guidelines also address Records 
Management issues which the Department’s policy does not. 

The major difference regarding Headers is that the Federal Guidelines specify 
that Headers and page titles (i.e. HTML page titles) should be the same. The 
Department’s policy does not indicate anything of this nature. 

The Federal Web Consortium Guidelines address the Body of documents. 
The Department’s policy does not include a similar section. 

The Department’s policy indicates that initials with a mailto: link are 
appropriate for the Standard Footer, but the Federal Web Guidelines clearly 
state that a (visible) e-mail address should be present. 

The Department’s policy indicates that documents greater than ten screens 
should be broken down into sections. The Federal Web Consortium 
Guidelines specify that documents longer than five pages should be broken 
down into sections. Clearly the Federal Web Consortium prefers documents 
that are shorter. 

The Federal Web Consortium Guidelines specifically address the issue of 
Cookies. The Department’s policy addresses cookies in only a very minor 
way, and does not include any example notices, even though reference is 
made to such notices. It is also not clear at this time whether the 
Department’s web site actually uses any cookies. 

The Federal Web Consortium Guidelines clearly place a much greater 
emphasis on collection of information from users. 



• Both policies address copyright, but the Federal Web Consortium Guidelines 
address special issues with regard to copyright that arise in the digital 
environment - the Department includes no mention of such issues. 

• Both policies also address Disclaimers, but there are a number of differences 
between the two policies. The Federal Web Consortium includes a 
Disclaimer of Liability, no such disclaimer is mentioned in the Department s 
policy. The Federal Web Guidelines include example disclaimer notices right 
in the policy document, but the Department’s disclaimer notices are located 
elsewhere and merely linked to which may make it difficult for users to find. 
The Department’s web policy documents were found to be inconsistent and 
contradictory with regard to the topic of Disclaimers. 

• The Federal Web Consortium Guidelines include a mention of Electronic 
Public Disclosure — the Department’s policy does not contain a similar 
section. 

• Both policies address GILS, but the Department’s GILS policy is rather 
vague. The Federal Web Consortium Guidelines clearly state that a GILS 
record should be created for each agency web site, but the Department s 
policy on GILS does not specify when GILS records should be created. 

Overall, the Department of Education websites comply with the guidelines offered in the 
Federal Web Consortium Guidelines. 

To a large degree, the Consortium Guidelines incorporated much directly from the 
Department of Education’s policies. Some of the major differences between the 
Consortium Guidelines and the Department of Education policy are that the Consortium 
Guidelines suggest that the creation of web policies should include a broad range of 
individuals other than the agency webmaster; the Consortium Guidelines include a 
concise and useful checklist for reviewing sites and pages to insure their compliance with 
policy; the Consortium guidelines (in its Appendix C) may have a greater emphasis on 
security issues; and the Consortium Guidelines have greater discussion of cookies and 
collection of personal information and disclaimers than the Department’s policy. The 
Department of Education’s policy has better examples of how to choose which texts 
should be linked to other websites. Detailed discussion of the findings can be found in 
Appendix V-5. 



Method 

As the Federal Web Consortium Guidelines were originally based on the primary 
Department of Education web policy document (C/.S. Department of Education World 
Wide Web Server Policy and Procedures at http://inet.ed.gov/intemal/wwwstds.html), 
these two documents are fairly straight-forward to compare in a side-by-side manner. 



While the actual structure and contents of each document are somewhat different, the 
headings and contents of the paragraphs in each section are similar enough to allow the 
analyst to match up the documents heading by heading. The study team used the side-by- 
side methodology. The team chose to limit the analysis to selected sections where the 
policies differed markedly. 

The similarity between the Department of Education policy documents and the 
Federal Web Consortium Guidelines limits the usefulness of this methodology since 
much of the Consortium Guidelines came originally from those developed at the 
Department of Education. Thus, other Federal agencies will find this side by side 
analysis to be more useful than the Department of Education - since other agencies 
policies are not so closely linked to those developed by the Consortium. Detailed 
findings from the side by side analysis comparing Department of Education policy to 
Consortium guidelines appears in Appendix III-5. 



Issues and Options 

The third revision of the Federal Web Consortium guidelines will soon be 
released. A key issue, which related to findings from other analyses presented in this 
section, is the degree to which the Department of Education should revise, expand, 
update, and better organize their various web policies. Detailed suggestions for such a 
revision to improve Department web policies can be found in Appendix III-5 and other 
appendices to this section of the report. 

Another issue and option related to the possibility of producing a single (or at 
least fewer) and better integrated set of Department web policy statements. One option 
for the Department of Education webmasters and/or the IWG is to review the newest 
Federal Web Consortium guidelines once they are released (the side by side used in this 
analysis is a November 1998 draft) and compare the content and organization to the 
Department’s guidelines, updating and revising Department policies where appropriate. 



DEPARTMENT WEB POLICIES IN THE BROADER 
FEDERAL INFORMATION POLICY SYSTEM 

A final aspect of the policy analysis was a preliminary comparison of the existing 
web policy system at the Department to (1) other selected departments and agencies, ^d 
(2) government-wide information policies that may have an affect on web policy 
development at the Department. This section of the report is not comprehensive and 
intends only to identify selected policy issues from other Federal agencies and those that 
cut across all agencies as a beginning point for further discussions at the Departnient 
regarding web-policy development. Although this assessment was not part of the original 
statement of work, the study team believes that officials might find this discussion of 
interest for future web policy development activities at the Department of Education. 




69 



55 



• The Copyright Act (17 USC 101) which provides certain protections 
for authors of literary and other t 5 rpes of work and sets the stage for 
determining intellectual property rights; it also allows government 
information to be free of copyright and encourages broad 
dissemination of government information. 

• Executive Order 12862, Setting Customer Service Standards 
(September 11, 1993). This Executive Order (and follow-up 
memorandums, e.g. Memorandum for Heads of Executive 
Departments and Agencies on “Improving Customer Service 
(March 23, 1995)) stresses the importance of developing customer 
service standards, benchmarking customer service standards, 
obtaining customer service data and assessments, and generally 
stressing the importance of “listening to the customer” as a means 
of developing and improving government services. 

• Government Performance and Results Act of 1993 (GPRA) (P .L. 103- 
62). This Act requires agencies to develop strategic plans with 
results-oriented objectives, performance measures and regularly 
report to Congress on their success in meeting the various 
performance measures. The Act is intended to improve Federal 
program effectiveness and improve public accountabiUty by 
focusing on program results and customer satisfaction. 

• The Paperwork Reduction Act of 1995 (P.L. 104-13), and a range of 
Office of Management and Budget, Office of Information and 
Regulatory Affairs (OMB-OIRA) circulars and bulletins, e.g.. 
Circular A-130, "The Management of Federal Information 
Resources;" these laws and the resulting policies describe how 
agencies are to manage information and otherwise make the 
information available to the public. 

• Information Technology Management Reform Act of 1996 (ITMRA) (P.L. 
104-106). This Act repealed the Brooks Act (P.L. 89-306) and introduced 
sweeping changes to Federal IT management by establishing agency CIOs and 
calling for agencies to better manage IT by improving the efficiency and 
effectiveness of IT in a range of government operations. 

• Executive Order 13011, Federal Information Technology (July 17, 1996). 
This Executive Order links the ITMRA, the PRA and the GPRA formalizes 
Office of Management and Budget (0MB) oversight of IT management and 
stresses the importance of perlormance-based planning and implementation of 
Federal IT. 
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• National Archives and Records Administration, GRS-20, Bulletin No. 98-02, 
Disposition of Electronic Records, and Electronic Records Working Group 
Report to the Archivist of the United States (September 14, 1998) (additional 
information and updates at www.nara.gov/records/grs20). Despite the 
ongoing litigation regarding the role of NARA in administering records 
management (including disposition and archiving, and preservation) agencies 
are required to schedule, archive, and dispose of electronic records - including 
information on websites. 

This list is not intended to be comprehensive. Rather it suggests a beginning set of 
Federal policy instmments that have some bearing on the development and management 
of the websites at the Department. 



Overview and Key Findings 

The policy system that has been developed for the management and development of 
websites at the Department of Education is very good. There is a basic web policy 
instmment. World Wide Web Server Policy and Procedures, and there is a suite of office 
and other web policy instmments that expand upon the basic policy and apply those 
policies to individual Departmental offices. Existing policies incorporate or recognize 
government-wide web-related policies. No other agency web policy system identified 
and reviewed was as comprehensive and intelligible as that at the Department of 
Education. Only in the area of policy for ongoing evaluation and compliance with GPRA 
did the study team identify weaknesses. Within this context, the study team offers some 
suggestions and options to improve selected policy topics covered in the Department’s 
web policy system. 



Method 

The study team compiled a set of (1) Federal agency web-related policy documents, 
and (2) selected Federal information policy instmments and guidelines with potential 
impact for web development and management. Members of the study team conducted a 
search through various agency websites in much the same manner that was used to 
identify Department of Education web policies (see discussion of this process earlier in 
this section). A limitation of this technique is that the compilation does not include 
agency web policy documents that may be in print format only and not posted to the 
website. The compilation of selected government-wide information policy that may be of 
interest for web development and management was based on the following criteria: 

• Was the policy instmment intended for a government-wide audience and not 
targeted at individual agencies? 

• Did the policy instmment mention web management/development directly or 
electronic information directly? 




71 



58 



• Did the policy instrument have a direct impact on web 
management/development in terms of public access to or use of Federal 
information? 

The study team organized the compilation of other agency web policies into a loose-leaf 
notebook that will be delivered as part of this project to the Project Liaison at the 
Department of Education as supplemental material to this report. Some considerable 
effort went into identifying and copying these web policies so that Department of 
Education officials might have an opportunity to know how other agencies are dealing 
with web policy issues. 

Members of the study team reviewed the two compilations (the first of all identified 
Department of Education web policies, the second of other agency web policies) in 
conducting the analysis described in this section. In the case of the compilation of other 
agency web policies, the team attempted to answer these questions: 

• How do the Department of Education web policies compare to others in term 
of length, scope, and completeness? 

• To what degree have other agencies dealt with issues and topics related to web 
management and development similar to those addressed by the Department 
of Education? 

Again, the study team cautions that the compilation of web policies is likely to be 
incomplete but that it does offer a useful listing and review of what other agencies are 
developing in the area of web policies. 

Regarding the review of the selected Federal information policies related to web 
development and management, the study team sought to identify issues and topics for the 
Department of Education to consider as it continues to evolve and refine its websites. 
Once again the study team cautions that this compilation and review is also likely to be 
incomplete of all the appropriate policy instruments that might be considered as having 
potential impact on Federal agency web policy development. In addition to reviewing the 
various compilations of policy statements, findings in this section also draw upon 
interviews with staff during the two visits to the Department of Education (see 
Management Assessment section for details on method). Key policy questions addressed 
in those sessions include: 

• To what degree do staff understand and are aware of various Federal 
information policies affecting Department website development and 
management? 

• How do staff stay informed of information policy developments that might 
affect Department web policy development? 

• What mechanisms are in place to update and modify existing Department _web 
policies in light of government-wide policy development? 
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have a well-developed mission statement and objectives, i.e., 
www.ed.gov/offices/OPE/OHEP/wwwplans.html . 

• There is some confusion and ambiguity as to policy development for apncy 
use and development of the Internet, hitranets, the web, electronic mail and 
related information systems (e.g., the policy by the Social Security 
Administration). Often times it is unclear how these various mechanisms for 
managing information systems are to work together. To some degree policy 
development for websites should be integrated into policy related to other 
systems. But an issue is the degree to which such integration is feasible given 
the rapidly changing environment of the web and other information systems. 

• The example web policies from other agencies have limited reference to or 
recognition of the broader policy instruments such as privacy, security, FOI, 
records management, GPRA, and others as listed earlier in this section. Using 
GPRA as an example (to be discussed below), there is little to no discussion in 
these policies of appropriate performance measures or other indicators of 
“success” for agency websites. Overall, there is much work yet to be 
accomplished in terms of integrating agency website policies with 
government-wide policies related to websites. 

• Despite the often-times complex and detailed information about developing 
and managing agency websites, there is relatively little offered about the 
importance of and need for training related to website development. When 
training is mentioned in the web policies, the guidelines are generally in the 
form of that as given in the Defense Technical Information Center’s policy, 
“DTIC staff are encouraged to obtain training and information necessary to 
implement their WWW projects.” 

• While such statements may offer encouragement, the experience of the study 
team is that most agencies are struggling with developing training programs to 
support the agency move to the web environment. Indeed, the Department of 
Education’s current policy on training related to web development may need 
to be reconsidered. Interviews with Departmental staff indicated a great 
desire on the part of most staff for more and better training related to web 
development and management, but a number of problems with obtaining 
support for such training appear to exist. 

A number of additional general findings related to ambiguous and unclear wording in the 
policies, confusion as to responsibilities for webmasters versus records managers, versus 
program officers, etc. can also be identified in these statements. But, the sample policies 
also offer some interesting approaches and ideas for policy development related to 
agency websites. For example the National Oceanic and Atmospheric Administration s 
(NOAA) “Report to the NOAA Deputy Under Secretary... on Internet Information 
Resources” raises a number of important issues. Officials at the Department of Education 
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may wish to review these policies in the supplemental loose-leaf notebook for a better 
sense of what other agencies provide for website policy guidance. 

Overall, the study team concludes that the Department of Education policy 
guidelines for web development and management are far superior to other agency website 
policies identified and reviewed. The U.S. Department of Education World Wide Web 
Server Policy ond Procedures fWAV^VSPP) (March 1, 1998) 

www.ed.gov/intemal/wwwstds.html is significantly more detailed, comprehensive, and 
intelligible than other policy guidelines identified from other agencies. The draft July 9, 
1998 “Proposal: Using Technology to Help American Education Improve and Innovate - 
and Invent the Future” is an excellent strategic planning and think-piece about moving 
forward in the web and networked environment. In addition, the suite of related web 
policy guidelines developed at the Department of Education and compiled in the 
supplemental loose-leaf notebook extends and supports the basic policies well beyond 
that identified in other agencies. Thus, the identification of areas for web policy 
improvement discussed earlier in this section should be understood in the context of the 
general high quality of the Department’s web policies - especially when compared to 
other Federal agencies. 



Recognizing and Integrating Broader Government-wide Policies 

An area that the Department of Education may wish to review is the degree to which 
existing Departmental web policies address the broader government-wide policies 
identified earlier in this section. Page 14 of the WWWSPP lists the various government- 
wide policies seen to be related to web development (which are quite similar to those 
listed earlier in this section. At issue is the degree to which specific content in these 
policy instruments should be better integrated into policy and procedures for web 
development and management at the Department of Education. This section identifies 
selected policy areas and discusses issues and possible options that the Department may 
wish to consider regarding the integration of those policy instruments into existing 
Departmental web policies and procedures. 

In the area of security the Department of Education Information Technology Security 
Manual (no date, but included in the supplemental loose-leaf notebook) provides a good 
example of developing policy that recognizes and is well-integrated into government- 
wide policy. The manual (which is apparently under development and/or evolving) 
clearly identifies a range of security issues and topics and discusses how they will be 
handled at the Department of Education. Of interest is the degree to which these policy 
guidelines are also integrated into the WWWSPP and the other suite of other web policies 
throughout the Department. It is not clear if the POCs and others responsible for web 
development in their particular office are aware of this manual and how it relates to web 
development and management. 

In the area of Privacy, Records Management, the Paperwork Reduction Act, and 
Freedom of Information Act (FOI), the OCIO website has developed a number of good 
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links and well-organized information about how best to deal with policy and procedures 
in these areas, see www.ed.gov/offices/OCIO/infocoll/info 1 .htnil . From this page 
customers can learn more about policies, Department of Education contacts, and other 
information related to these topics. Staff at the Department of Education can also obtain 
information regarding procedures to follow related to implementing these policy areas 
from these pages. The study team also determined that a search on the terms records 
management,” “Freedom of Information,” and “privacy” from the www.ed.g oy 
homepage “search key” would have identified these policy pages at OCIO as well as 

other webpages. 

But, for example, there are a number of excellent guides for how citizens might use 
the FOI and privacy acts. One such guide is A Citizen's Guide on Using The Freedom of 
Information Act And The Privacy Act of 1974 to Request Government Records (U.S. 
House of Representatives, Report 105-37). This guide provides specific instructions and 
example request letters and could be linked directly through the Thomas website at the 
Library of Congress or through other sources such as Electronic Privacy Information 
Center (EPIC) http://www.eDic.org/open gov/citizens guid e 97.html>. Another 
example are the links from the OCIO page to records management policy Md procedures. 
There is no indication that the existing GRS 20 is currently “null and void” nor are there 
additional links to records management policy, procedures and other information at 
www.nara.gov/records/grs2Q . At issue is the degree to which the Department of 
Education wishes to provide additional detail and linked information about these 
government-wide policy areas - perhaps as much for Departmental staff as customers. 

In the context of records management, it was not clear to the study team how items 
placed on Departmental websites were controlled. Policy for how long a document 
would remain on a website, under what conditions it would be removed, and how it could 
be determined when and for how long an item was on the website were not identified. 
During interviews and focus group sessions, there was some concern from participants as 
to the Department’s ability to go back and produce an official record that appeared at 
some point in the past on a website but no longer was posted. There also appeared to be 
some issues related to having an electronic version of a particular document on a website 
that might vary in some way from an official print version, e.g., the electronic version 
being updated and the print version not being updated. As per requirements in the 
Federal Records Act (44 U.S.C. Chapters 29, 31, 33, 35) and for legal/evidentiary 
requirements, policy and procedures in this area should be reviewed by the Department. 

In addressing these and related issues regarding electronic records management of 
Federal websites. Department officials may wish to refer to a recent report by McClure 
and Sprehe, Analysis and Development of Model Quality Guidelines for Electronic 
Records Management on State and Federal Websites (1998). Of special interest are the 

proposed guidelines in Chapter 6, http://istweb.svr.e d u/~mcclure/nhprc/nhprc_title.html . 

These guidelines and suggestions may assist the Department better develop policy and 
procedures in this area. 



The study team noticed that in the area of security and privacy there are few methods 
by which customers can submit information that is encrypted or otherwise secured. 
While it may be that currently there are few services on the website that require such 
levels of security and privacy, such are likely to be needed in the future as the 
Department, and the government overall, move to providing a broad range of personal 
services that will require secure transfer of data, personal identification numbers, 
electronic signatures, etc. The Department may want to begin thinking about policy 
initiatives in these areas. Regarding privacy. Department officials should review an 
excellent set of principles and suggestions offered by General Services Administration 
(GSA) at http://www.itpolicv.gsa.gov/mke/fedwebm/privacy.htm for possible changes 
that might be made in the various privacy statements presented throughout the 
Department’s websites. 

The study team could not locate an electronic reading room for declassified 
documents on the Department of Education website as per instructions in the Electronic 
Freedom of Information Act (1996). Perhaps such a page is available on the website and 
it simply could not be identified by the study team, or perhaps there are no declassified 
documents to be posted on such a site. Nonetheless, Departmental officials may wish to 
review this issue. In addition, the study team did not assess the website in terms of Year 
2000 (Y2K) compliance or policies related to Y2K as developed by the Department. If 
they have not already done so. Department of Education officials should also review the 
degree to which Y2K problems might affect the operation of the website as it approaches 
the millennium. 



Another issue concerns the degree to which the other office homepages, such as 
the OESE page www.ed.gov/offices/QESE. should have links to basic policy guidelines 
regarding privacy, records management, FOI, etc. Currently, on the OESE homepage, 
there is no link or information about such topics. While it is understood that on the 
www.ed.gov homepage there is a link in the top right comer to disclaimers and notices 
which will lead to a privacy statement, generally it is difficult to find the Departinent s 
policy on privacy. Further, the privacy statement as a “Disclaimers and Notices does 
not promote users’ awareness of their rights related to privacy. 

An option for the Department to consider is replacing the phase “Disclaimers and 
Notices” with a button labeled “Privacy and other Policies” on the main Ed homepage. 
That button could then provide a listing of the various policy documents and procedures 
available and provide links (e.g., to the OCIO page noted above). The Department ^§ht 
also have a similar button on all the office homepages for “Privacy and other Policies 
linking back to that ED homepage content. Thought might be given to expanding the 
policy content here include statements which are now elsewhere (in the WWWSPP) on 
topics such as the use of “cookies,” etc. 
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Current Awareness 
Management 



of Federal Policies that May Affect Web Development and 



Although there are appropriate references to and integration of government-wide 
policies that affect website development and management at the Department, interviews 
with staff suggest that there is some lack of understanding of the importance, use, and 
application of these policies. The study team especially noticed limited understanding of 
electronic records management issues, scheduling and archiving electronic records, and 
the importance of being able to know exactly what information, resources, tools, etc. 
were on which website at what particular point in time. The study team also noted limited 
attention to GPRA requirements (see discussion below). 

For example, the World Web Web home Page Guidelines and Best Practices 
(World Wide Web Consortium, November 1996) is currently under revision with a draft 
being circulated for review. Department of Education staff may wish to affect this policy 
development process by offering suggestions and comments. Some additional policy 
work is also under development in General Services Administration, one of which is draft 
policy related to Chargeback issues for shared Internet website operations (contact 
rich.kellet@gsa.gov) . Also under development by the National Partnership for 
Reinventing Government is “GovWeb” which is intended to be a website of Federal 
websites. 

Departmental officials may wish to consider having (1) some general training 
sessions for webmasters and the IWG on the importance and implementation of these 
policies, and/or (2) assign specific responsibilities to an individual or team to monitor 
Federal policies that may affect the website and determine the degree to which the 
Department is complying with these policies. 



Policy for Ongoing Evaluation of the Website and GPRA Compliance 

Regarding successful compliance with GPRA, the study team concludes that while 
some work has been accomplished regarding the development of performance measures 
for the website, additional development of measurable objectives, performance measures, 
and procedures for ongoing data collection, reporting and analysis is needed. The 
primary source for these objectives and performance measures appears to the U.S. 
Department of Education Strategic Plan: 1998-2000 Objective 4.1 (although other 
objectives may bear indirectly on web management and development). Apparently, the 
Department of Education has (as of March 18, 1998) not submitted a performance plan as 
per GPRA requirements (http://ombwatch.org/www/ombw/gpra/gprasp.html). 

Performance measures for website activities must stem from clearly stated goals and 
objectives. Such goals and objectives can be developed for both the main Departmental 
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homepage as well as websites for individual offices. In thinking about categories or 
types of performance measures for the website, the following may be useful: 

• Extensiveness. How much of what specific types of services the website 
provides (e.g., number of users accessing a Web page per week, number of 
remote dial-ins per week); 

• Efficiency. The use of resources in providing or accessing networked 
information services (e.g., cost per session in providing access to web 
customers, or the average time required to make a correction or other type of 
change to a particular website); 

• Effectiveness. How well the web information service met the objectives of the 
provider or the customer (e.g., success rate of identifying and accessing the 
information needed by the user); 

• Service quality. How well a service or activity is provided by the website 
(e.g., percentage of transactions in which users acquire the information they 
need); 

• Impact. How a service made a difference in some other activity or situation 
(e.g., the degree to which web customers enhanced their ability to teach a 
secondary class better); 

• Usefulness. The degree to which the web services are useful or appropriate 
for specific types of customers (e.g., percentage of services of interest to 
different types of user audiences); and 

• Adoption. The extent to which institutions such as schools or users integrate 
and adopt Department of Education resources or services into organizational 
or individual activities (e.g., answering reference questions at the school 
media library, generating lesson plans, or writing proposals). 

These categories suggest a range of possible performance measures that can be developed 
for both the overall Department homepage as well as individual office websites. For a 
range of reasons, costs and benefits related to the Department’ s website are needed. For 
example, the National Science Foundation claims that they saved $750,000 in printing 
and mailing costs by posting its Grant Proposal Guidelines on their website (government 
Computer News, October 26, 1998, p. 32). 
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Once performance measures for the website are identified and defined, it may be 
useful to develop quality standards for each of the measures. For example, the 
“performance indicator” for objective 4.1 (no. 1) in the Strategic Plan (p. 45) is: 

By 2001 at least 90% of customers, internal and external, will agree that ED 

products, services, and information, including those on the Department s website, 

are of high quality, timely, and accessible. 

In fact, this “performance indicator” is a quality standard. The performance measure 
being used in this example is “user satisfaction.” The standard that the Department has 
set as acceptable is 90% satisfaction with ED products, services, and information. But 
the proposed quality standard will be difficult to measure because, first, it really includes 
nine different measures: 

• Product quality 

• Product timeliness 

• Product accessibility 

• Service quality 

• Service timeliness 

• Service accessibility 

• Information quality 

• Information timeliness 

• Information accessibility. 

Second, it is unclear how each of these measures will be operationalized so that data can 
be collected to produce the measures. Third, it is not clear who has responsibility for the 
data collection and reporting. 

The Department of Education may wish to consider how it might best improve its 
compliance with GPRA for its website. A process needs to be developed that 1) clarifies 
website goals and objectives for both the main homepage as well as individual offices, 
and identifies, defines, and operationalizes performance measures and quality standards 
for the website; 2) establishes a means to collect, analyze and report data to produce the 
performance measures; 3) determines the degree to which the quality standards are being 
met; and 4) then integrates these findings into an ongoing process to improve the overall 
quality of the Department’s websites. Additional detail about how to develop and submit 
performance plans can be found in the Act itself (Sections 1 15 and 1 16). 

Better use of existing log server data and perhaps other existing web-based data such 
as customer surveys may assist in developing this process. Existing contractors might be 
tasked specifically to produce the data needed for the performance measures and quality 
standards once these measures and standards are agreed upon. The IWG, the OCIO, or 
the Webmaster at OERI appear to be candidates to have formal responsibility for 
administering a process of ongoing evaluation. To some extent automated data collection 
that regularly collects information on web activities and use can be linked directly to 
producing the performance measures and quality standards. Regardless of the process 
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chosen, the Department of Education needs to better develop and define an ongoing 
means to evaluate its website, inform webmasters and the IWG of the results of such an 
evaluation, and comply with requirements of GPRA. 

OVERVIEW OF THE DEPARTMENT’S WEB POLICY SYSTEM 

The Department of Education has a significant amount of web policies. Indeed, a 
next step for the Department to consider is to better organize, coordinate, and perhaps 
make more concise a single source where all website policies can be found. The study 
team invested considerable time identifying and locating the various web policies that 
have been developed (see Figure III-2 and supplemental loose-leaf notebooks). 
Coordinating these various policies may be difficult if they are all not easily identified, if 
they are not linked, or if they grow and evolve without central coordination and review. 

The study team employed a number of different assessment techniques to evaluate the 
policy system currently directing website development and management at the 
Department of Education, these included: 

• Identification and location of web policies 

• Analysis of Department of Education Policies 

• Analysis of the Implementation of Web Policies 

• Comparison of Department of Education Web Policies to Guidelines 
Developed by the World Wide Web Consortium 

• Review of Selected Web Policies by other Agencies 

• Recognition and Integration of Selected Government-wide Policies into 
Department of Education Web Policies. 

Overall, the study team finds that the Department of Education has developed an 
excellent policy basis for the management and operation of the Department’s websites. 

Policy development for electronic records management and ongoing web 
evaluation and compliance with GPRA requires additional attention. Additional attention 
might be given to simplifying, coordinating, and organizing the various policies into a 
more unified whole. Nonetheless, those individuals who have been developing the web 
policy system at the Department of Education should be congratulated for their work in 
this area. They should also be encouraged to continue to develop, coordinate, link and 
update these policies as outlined in the various recommendations throughout this section 
and detailed in the appendices. 
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IV: TRANSACTION LOG AND TECHNICAL ANALYSIS 



OVERVIEW AND KEY FINDINGS 

Technology that aids in the management of websites is changing very rapidly, 
making it difficult for website administrators to stay abreast. Two years ago, most 
organizations built websites largely to disseminate information. Traditional 
“webmasters” added HTML pages or edited existing pages within their site to add to the 
information content. Websites that began with ten, twenty, or even one hundred pages 
have grown exponentially, in some cases reaching several thousand pages. Today web 
administrators have more sophisticated jobs. In order to determine technological 
advancements, assess target audiences, and improve overall site quality, they must 
collect, analyze, store, and manage vast amounts of information collected about site 
usage. 



Log file analysis and other related technical analyses provide a method for web 
administrators to collect, analyze, store and manage website usage data. The study team 
used log file analysis and several other related technical analyses as one of four 
assessment approaches for the evaluation of the Department website. 

The study team undertook analysis of the Department of Education log files in 
order to answer the following questions: 

• How many broken links exist within the Department website? 

• To what degree do Department web pages meet effective HTML coding 
practices? 

• To what degree do Department web pages follow effective design practices? 

• What paths do users take through the first few layers of the website? 

• What areas of the homepage do users most often select? 

• To what degree do external search engines find and retrieve Department web 
pages? 

• To what degree did users find links produced by Department’s search engine 
relevant? 

• Do current query forms support GET and POST query methods? 

• How frequently are current log reports utilized? 

Several key finding emerged from this transaction log and technical analysis: 

• Broken links exist within the Department website. 

• Coding of some pages could be improved. 

• Department web pages have inconsistent design practices. 

• Path analysis highlighted both popular and underused routes through ihe first 
several layers of the site. 
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• Users on the homepage most often select links on the vertical navigation bar 
or the Most Requested Items section. They do not select links from the 
horizontal navigation bar. 

• Some external search engines do not effectively find and retrieve Department 
web pages. 

• Study team could not determine to what extent users found links produced by 
the Department’s search engine relevant due to current Department query 
practices. 

• Some query forms can support the GET query method. 

• Several methods exist for increasing the effectiveness of current log reports. 

While the study team identified the problems above, it is important to note that the 
Department is, itself, making use of transaction log analysis software and has been 
involved in an ongoing process of monitoring the errors and resolving them as frequently 
as possible. The study team’s findings suggest that these efforts should continue and be 

expanded. 

This section continues by providing background about log file analysis and 
current Department log practices, reviewing the methodology which the study team 
employed, discussing the findings of the data collection and suggesting several options 
for improved website management. 



BACKGROUND FOR LOG ANALYSIS 

This section first provides an overview of log analysis followed by specific 
procedures used in the analysis, findings, and options. Some readers may be familiar 
with the information in the first subsection; the research team asks their indulgence. 



Log analysis is a three-step process that involves the set-up, analysis, and 
interpretation of data. It is important for web administrators to: 

1 . Determine what information administrators want to log. Most will find that it 
is not necessary to log all information supported by today’s web servers. 

2. Develop or purchase the necessary software, capable of manipulating and 
presenting relevant information to web administrators. Appendix IV- 1 is a 
current review of the latest log analysis software. 

3. Analyze reports. Log analysis software is capable of producing lengthy 
reports with numerous charts and graphs. Running this software on a weekly 
or monthly basis is only half the task. Adimnistrators must carefully analyze 
the information supplied and then take the necessary steps to continue 
managing their web pages and web server. 
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Log files are text files that can range in size from IMB to 100s of MB, depending 
on the traffic at a given website and server configuration. In order to detenrune and 
report on the traffic that a website receives, it is important to understand the difference 
among some key log file terminology. There are significant differences between a hit, 
access, visit, and visitor. 

A hit is any file from a website that a user downloads. A hit can be a text 
document, image, movie, or a sound file. If a user downloads a web page that has 
six images on it, then he or she puts seven “hits” on the server (six images + one 
text file). The number of images on a website varies greatly, so this type of 
statistical reporting can be misleading. 

An access, sometimes called a page view, is an entire page downloaded by a user 
regardless of the number of elements on the page. If a user downloads a web 
page that has six images on it, the user puts one page access on the server. 

A visit is a user session with a website, regardless of the number of pages or 
elements he or she viewed. If a user looked at sixteen pages and fifty-four 
graphics while at a website, that user put one visit on the web server. A visit is 
determined by a user’s IP address, which can be misleading due to Internet 
Service Providers (ISPs and Firewalls or Proxy Servers). A visit becomes a 
different visit after 30 minutes of inactivity time. For example, a user with the IP 
address 129.230.33.90 was browsing the Department of Education website 
between the hours of 12:00 pm — 12:10 pm. At 12:15 pm this user goes to lunch, 
comes back at 1:10 pm, and continues to browse the Department of Education 
website. Because this user has been inactive for more then 30 minutes, the user s 
activity will be considered two visits. 

A visitor is a unique IP address in the log over a given reporting period. In the 
above example, the same IP address accounted for two visits, but would only be 
considered a single visitor. 

It is important to use log analysis software that has the capability to report all four of the 
above statistics. 



Access log 

Historically the access log is the most commonly analyzed log file and provides 
the greatest amount of server data, including the date, time, IP address (or domain name), 
and user action (i.e., document/imagesAiomepage.gif). The following is an example line 
of text from an access log: 

128.230.33.90 - - [25/Feb/1998:-2:38:09 -0400] “Get /library/index.html” 200 1872 
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From this line of code it is possible to analyze the following variables in the access log. 

1. IP address or domain name of the user’s system. In the above example, the 
user was browsing the web from the IP address 128.230.33.90 
(headcase.syr.edu). It is important to note that ISPs dynamically assi^ IP 
addresses to their users, maldng it harder to differentiate between individual 
users. 

2. Date and Time. In the above example, the user accessed the web page on 
February 25, 1998, at 2:38 AM and 09 seconds. By default the time is based 
on a 24-hour clock. The last part of the time stamp (-0400) specifies the offset 
from Greenwich Mean Time (GMT). This could be useful when trying to 
match up the log files from a large website with mirror sites in different time 

zones. 

3. Item accessed. The word item can mean an image, movie, sound, HTML, or 
CGI file. The above example shows that index.html was the item accessed. It 
is also important to note that the full path name (from document root) is given 
to avoid confusion (i.e., there may be more then one file called index. html on 
the web server). 

4. Status code. The status code tells if the user had a successful request or some 
sort of error. In the above example the 200 means that the user had a 
successful request. In general, a status code in the 200s indicates a successful 
download, a status code in the 300s indicates that the user was redirected to 
another page, and a status code in the 400s or 500s indicates that the user 
encountered some sort of error. 

5. Bytes Transferred. In the above example, 1872 refers to the number of bytes 
transferred between the web server and the client computer. 

It is possible to generate the following data from these variables: 

1. Xhe percentage of users accessing the website from a specific top-level domain 
(i.e., .com, .edu, .mil, .org, .net, .gov). This can then be broken down into the 
number of hits, accesses, or visits from specific sub-domains (e.g., 
www.microsoft.com or www.syr.edu). 

2. The overall breakdown of hits, accesses, and visits. 

3. The number of requests the web server receives during specific hours and days of 
the week. These statistics can be useful to network administrators who need to 
know the optimal time or day to perform server maintenance and/or system 
upgrades. 
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4. Most and least requested pages within the website. This can be useful to 
determine which parts of a website are successful and what pages may need 
further attention. 

5. Entry and Exit pages. This can help a web administrator determine what pages 
users may be bookmaking (entry page) and what pages may need a redesign due 
to large file size, broken links, or sub-standard content (exit page). 

6. Path analysis. This can show an Administrator what the most common path(s) 
users are taking through a website. This information can be useful in redesign 
and overall maintenance of a website. 

7. Demographics. This can be used to indicate what country is accessing the 

website, even down to specific cities and states. It is important to note that this 
information is taken from the Intemic’s WHOIS database and can be misleading 
(e.g., all America Online users appear to come from Vienna, Virginia). 

The information that can be generated from the access log provides a broad view of a 
web server’s use and users. Analysis of this information enables server administrators to 
characterize their server’s audience and usage patterns. It is important to point out, 
however, that pages may be cached on a client’s machine, thus the log will under- 
represent some usage of the site. 



Agent Log 

The agent log provides data on a user’s browser, version of browser, and operating 
system. Today web pages are designed with a specific technology (Java, ActiveX) and 
web browser in mind. With the ever-growing browser war between Netscape and 
Microsoft, it is important to know what browser and operating system users are accessing 
a website with so developers know how to best maintain their site. Following is a sample 
agent log entry: 



Mozilla/4.04 (Win95: 1) 



Analysis of the agent log allows server adimnistrators to determine the following. 

1. Browser. The type of browser used to access a website is important since there 
are several different web browsers on the market today (i.e., Netscape 
Communicator, Microsoft Internet Explorer, LYNX). Each of these support 
different technologies and are programmed to view web pages differently. 

2. Browser Version. Website developers need to know when it is time to begin 
using a new and innovative technology (JAVA, Shockwave). For example, 
earlier versions of Internet Explorer do not support many JavaScript functions. 
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3. Operating System. The type of computer and operating system users have can 
determine the way a website will look (i.e., Windows NT, Windows 95, MAC 
OS, or UNIX). 

These analyses are essential for the design and maintenance of websites. Without this 
information, site administrators could design sites that require viewing capabilities that a 
vast majority of the site’s users do not possess. At best, this leads to wasted effort by the 
server administrator. At worst, this can lead to improperly displayed web content, thus 
effectively rendering the site inaccessible to the user. 



Referrer log 

The referrer log indicates when one site on the web links to another site on the web. 
Each time a user clicks on a link from one site to another, a referral line is added to the 
log. A sample entry is as follows: 

http://altavista.digital.com/cgi-bin/query?pg=q&what=web&kl=XX&q=Technology > 

/tech/technology.html 

In this example, a user searched for the term technology using Altavista’ s search engine, 
and Altavista is linking to a file named technology.html. Analysis of the referral log 
allows administrators to track the following: 

1. Referral. If a user is on a site (i.e., www.syr.edu), and clicks on a link to another 
site (i.e., www.lib.de.us/), then the www.lib.de.us site will receive an entry in their 
referral log, showing that a user clicked from syr.edu to lib.de.us. 

2. Linked Page. The referrer log also shows administrators the exact page that is 
being linked to. This information can prove to be important to website 
administrators when a site is undergoing a redesign. 

Tracking and analyzing referral data is critical for alleviating incorrect links (error 
404). For example, when the URL of a page within a site changed, the site administrator 
would notify all referrals of the change. Doing this will alleviate the problem of “Error 
404 - File Not Found’’ messages. 



Error log 

All web users have encountered the message “Error 404 - File Not Found” while 
browsing the web. This message indicates that a particular page that the user was looking 
for does not exist. Every time this message appears for a user, it also is generating a line 
of code in a web server’s error log. 
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[Mon Feb 16 18:25:04 1998] access to /da-srvcs.html failed for 128.230.33.134, reason: 
File does not exist. 

The error log contains the following data for analysis: 

1. Error 404 - As shown in the above example, the error log tells a website 
administrator the date and time, IP address of the user’s computer, page on which 
the error was received, and the type of error generated. These error messages are 
critical for website administrators to analyze, as they inform administrators of 
unavailable links within a site. 

2. Stopped Transmission. Every time a user hits the stop button on a web browser, a 
stopped transmission error is generated in the error log. As in the error 404 
message, the stopped transmission error tells a website administrator the IP 
address, specific page, and the error type. Analyzing this information can indicate 
pages that are too large (usually due to large graphics or slow CGI scripts). 

The analysis of the error log can provide important server information to the website 
administrator. This information can make it easier for site administrators to modify and 
correct web content, thus decreasing the number of errors users encounter while 
navigating a site. 



Log File Formats 

The most popular web servers today run on a UNIX or Windows NT server (mainly 
due to their security). The majority of web servers use the same standard when tracking 
log files, the W3C standard. The W3C standard delimitates fields within the log files by 
spaces so that the order in which the log file information tracked is standardized. 
However, within the W3C most web servers support two standard log formats. 

1. Common Log Format. This format uses four log files (access, agent, referrer, Md 
error) to track user information. Although all web servers are capable of tracking 
the four log files, some are set up by default to only track one or two (usually the 
access and error). 

2. Extended Log Format (also referred to as the flexible log format). The extended 
log format combines all log information into one file. Using this format allows 
website administrators to track all possible information about a single user. Most 
web server’s today are capable of tracking log files in the extended log format, 
however, a site administrator may need to change the necessary configuration 
files. 
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Department of Education Web Server Logs 

The Department of Education has several web servers within the ed.gov dom^n. 
On a tvpical day www.ed.gov collects nearly 100 MB worth of log files. Due to the size 
of the^g files, the study team received one week of log files (dated September 2 
September 26, 1998) from www.ed.gov. In addition, the team collected 
files from search.ed.gov files (dated September 20 - September 26, 1998), and two 
weeks of log files that were specific to OESE (dated September 13 - September 26 
1998). The Department uses the Netscape Enterpnse 3.5.1 web server for its mai 

website. 

The web server is configured to use the extended (or flexible) log file format, 
which combines access, agent, and referral data into one log file (the error ata is 
contained within a separate file). Search.ed.gov is using an UltraSeek ®"Sine 

which generates two sets of log files. Although there are standard log files foat are 
collected for search.ed.gov, there are also separate log files that are crewed which are 
specific to a user’s search. For this analysis, study team collected the W3C standard log 

files. 

The Department of Education uses WebTrends log analysis software to regularly 
run detailed reports for offices within the Department of Education and an ovwall report 
for the entire ed.gov domain. Reports are distributed on a monthly basis. The current 
mondi’s statistic: are cumulated daily. The WebTrends reports can be found at: 
http:/Avww. ed. gov/intemalAvebstats/ 



METHOD 

The study team formulated these questions after consultation with Keith Stubbs. 
Smbbs requested that the smdy team concentrate on an advanced as the Department w^ 
already running some basic reports on a regular basis. This section ^ 

describing the testing environment and by explaining how the smdy team gathered 
about each of these questions. 



Test Environment 

On September 25, 1998, the study team set-up an FTP account for the Department 
of Education on a UNIX server located at Syracuse University^ 

FTP’d over approximately 15 compressed log files. Between Septembw 20-26 , 

smdy team collected approximately 800 MB of log data from the 
Education. Team members then uncompressed and transferred Sr 

system. The system consisted of a Pentium H, 300 MHz Microsoft , 

with 256 MB of RAM, and installed with Microsoft SQL Server 6.5, WebTre 
Enterprise Edition 2.1, and Microsoft Site Server 3.0 Commerce edition. 



Questions Addressed 



Each question addressed by the log analysis used different techniques. These 
techniques are outlined below by question. 

How many broken links exist within the Department website? 

The study team ran a standard Site Server report that showed any broken link 
within the Department website. Broken links consist of web links that do not work, and 
result in a 404 “file not found” user error. It is important to note that 404 messages may 
occur for several reasons; links that do not work, mistyped URL’s, and URL’s that ^e 
incorrect on remote sites linking to www.ed.gov. The Department can work to imniimze 
404 messages due to missing files but may not be able to reduce the number of these 
messages to zero given the other causes of 404 messages. 

To what degree do Department web pages meet ef f ective HTML coding practices ? 

The study team created a custom filter to search for excessive image size. The 
team defined excessive image size as any image over 70 Kb. After identifying these 
images, the team tested how many large images stemmed from improper height and 
width attributes within the HTML <img> tag. Proper height and width attnbutes within 
an HTML file should match the actual height and width of the image. 

To what degree do Department web pages follow effect ive design practices? 

The study team created custom filters to search for excessive image size and 
excessive HTML page size. The team defined excessive image size as any image over 70 
Kb and excessive HTML page size as any web page (excluding images) exceeding 70 

Kb. 

What paths do users take through the first few lay ers of the website? 

Using Microsoft Site Server, the study team created custom reports that detailed 
paths that users took through the Department of Education website. Site Server contains 
a built-in path report, which reports the top five user paths. Because of the large number 
of links on the Department homepage, the study team felt it necessary to collect data on 
the top ten user paths. The research team created a custom filter within Site Server to 
accomplish this. 

What areas of the homepage do users most often select? 

The study team mapped the URLs given in the above described user path report to 
physical locations on the Department web page. The study team broke the Department 
homepage into four major areas, illustrated in Figure IV- 1 
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Figure IV-1: Department Homepage Key Link Areas 




1: Left-hand navigation bar 
2: Education Headlines links 
3: Most Requested Items links 
4: Horizont^ navigation bar 

The research team wanted to determine what areas of the homepage users most often 
selected. 

To what degree do external search engines find an d retrieve Department web pages ^ 

The study team experimented with three separate search engines in order to assess 
the direct accessibility of Department resources from external sources. The study teani 
chose two external search engines known for their specialization in locating feder^ 
government information: Pathway Indexer and GovBot. In addition, the study team use 
AltaVista, a popular, commercial search engine. 

Pathway Indexer http://gather.access.gpo.gov/Harvest/brokers/Pathway/ 

GovBot http://ciir2.cs.umass.edu/Govbot/ 

AltaVista http://www.altavista.com 

A study team member browsed the Department website and selected ten words or 
phrases reflecting actual content on the site. The investigators entered each term or 
phrase into each search engine. These terms were: National Center for EducaUon 






Statistics, ERIC digests, Education headlines. Education, Department of Education, 
Education reform. Goals 2000, Adult education. Educational resources, and Digest of 
Education Statistics. The investigators examined the degree to which the query results 
included Department of Education links. The investigators defined successful searches as 
those that produced a Department of Education link within the first page of query results. 

It is important to note that each search engine lists a different number of query 
results on its first page. GovBot and AltaVista list 10 links per query results page. 
Pathway Indexer however, lists all relevant links on its query results page. Therefore, tne 
Pathway Indexer had a greater chance of success than the other two search engines. 

There are other strategies that might be employed to explore the question of 
effective retrieval including use of the reports produced by WebTrends on refemng sites. 
Due to the limited scope of this project, those reports and other techniques could not be 

employed. 

To what degree did users find links produced by Depa r tment’s search engine relevant ? 

The study team analyzed a random sample of users drawn from one week of log 
data from the search engine log. The study team wanted to determine how many pages of 
search results the average user went through before choosing a link (there are ten results 

per page). 

In order to obtain a random sample, the study team split the 130,000 line search 
log into 130 mini log files, each containing 1000 lines. The study team took the first IP 
address found within the log file and counted the number of search result pages the IP 
address browsed through, prior to link selection. Departmental usage of the POST query 
method limited the study team’ ability to determine what query terms users searched. 
Therefore, the study team will not present findings at this time. 

Do current nuerv forms support GET and POST query methods? 

Staff working for Keith Stubbs gave the study team telnet access to the 
Department of Education search engine server in order to conduct several interactive live 
tests. Study team members wanted to determine to what extent the Department website 
could support the GET query method. The study team created two identical forms, one 
that used the GET query method, and the other that used the POST query method. The 
investigator entered the term “drug programs” into both forms and ran each query. The 
researcher captured the results in text file for later analysis. 
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How frequently are current log reports utilized? 



The study team wished to determine to what extent the Department of Education 
made use of currently existing log reports. To do this, the study team requested that 
members of Keith Stubbs staff run a report about current log report usage. The report 
shows all accesses to the Department of Education October WebTrends reports from 
November 1 - 15, 1998. The study team counted the number of unique Department of 
Education IP addresses that accessed the files in this period. 



FINDINGS 



WebTrends Reports 

The Department of Education uses the commercial software WebTrends to run 
detailed log reports. WebTrends is an excellent application for parsing through large 
amounts of log data. The WebTrends reports indicate an active site with over 3.5 million 
accesses per month, nearly 1 million of which constituted separate visits. However, such 
a large site is not without its problems. The following subsections discuss several key 

findings. 

How many broken links exist within the Department website? 

There are thousands of instances in which the Department of Education website is 
linking to unavailable resources, also known as 404 errors. The longer a website exists 
and the larger (in number of pages) a website grows, the more susceptible it becomes to 
404 errors. Over time, web pages are redesigned, causing path or file names to change, 
or pages to be deleted. These factors lead to links becoming unavailable to users. 

The Department of Education website contains three types of 404 errors: 

• Internal - Where the Department of Education is linking to an unavailable resource 
within the main ed.gov web server. 

• Different Internal Server - Where the Department of Education is linking to an 
unavailable resource outside of the main ed.gov web server. 

• External - Where the Department of Education is linking to an unavailable resource 
on a server outside of its domain. 

The Department monitors 404 errors on a somewhat regular basis and makes efforts to 
correct these problems through fixing URL’s, page redirects, and emailing other sites that 
use incorrect URL’s. According to Keith Stubbs, the number of 404 errors dropped 25% 
from September 1998 to October 1998 despite a rise in page views. 
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Table IV- 1 IV-2, and IV-3 provide examples of the three types of 404 errors that 
a user may encounter while on the Department site. These are exclusive of 404 errors a 
user may receive because he or she mistyped a URL or a remote site not J^^der E 
control provided an incorrect URL. There are three columns to each table below, -^e 
first column identifies the URL that is unavailable. The second column shows all pages 
from within the ed.gov website that are attempting to link to the unavailable resource^ 
The third column shows the hypertext link that a user would see on the page 
the unavailable resource. These examples do not represent all 404 errors on the site, u 

give a flavor of them. 



Table IV-1: Internal errors within www.ed.gov 



Unavailable URL 


Link to URL 


Hyperlink Text 


1. http://www.ed.gov/Ma 
ilingLists/ED Info/0 1 2 
9.html 


http://www.ed.gov/faqs.ht 

ml 


EDInfo Information 
Service Listserv 




http://www.ed.gOv/tfaqs.h 

tml 


EDInfo Information 
Service Listserv 


2. http://www.ed.gov/Se 
arch/ 


http://www.ed.gOv/pubs/E 

DInitiatives/98/98-02- 

20.html 


http://www.ed.gov/Search/ 




http://www.ed.gov/PressR 

eleases/02- 

1998/redesign.html 


http://www.ed.gov/Search/ 




http://www.ed.gov/Mailin 
gLists/EDInfo/msg003 3 8 . 
html 


http://www.ed.gov/Search/ 




http://www.ed.gov/Mailin 

gLists/EDInfo/msg00334. 

html 


http://www.ed.gov/Search/ 




http://www.ed.gov/offices 

/OCIO/welcome/twel26.ht 

ml 


http://www.ed.gov/Search/ 


3. http://www.ed.gov/pu 
bs/PFIE/employ/sld03 
O.html 


http://www.ed.gOv/pubs/P 

Fl^employ/index.html 


The Promise Fulfilled 




http://www.ed.gOv/pubs/P 
FlE/employ/sldOO 1 .html 


Last Button 




All the slides in this 
presentation 


Last Button 



Table IV-2: Internal errors within different ed.gov servers 



Unavailable URL T 

1. http://easi.ed.gov/html 

/faqs.html 


Link to URL 

http://www.ed.gov/faqs.ht 

ml 


Hyperlink Text 
Project EASI 




http://www.ed.gOv/tfaqs.h 

tml 


Project EASI 


2. http://ocfo.ed.gov/gmt 
info/reen.htm 


http://www.ed.gov/tfundin 

g.html 


Discretionary Grants 
Redesign 


3. http://nces.ed.gov/nces 
NewPubs.html 


http://www.ed.gOv/pubs/E 

DInitiatives/96/09-20.html 


" ittp://nces.ed.gov/ncesNe 
wl^bs.html 




http://www.ed.gOv/pubs/E 

DInitiatives/96/07-09.html 


(http://nces.ed.gov/ncesNe 

wPubs.html) 




http://www.ed.gOv/pubs/E 
DInitiati ves/96/07- 1 7 .html 


(http://nces.ed.gov/ncesNe 

wPubs.html) 




http://www.ed.gOv/pubs/E 
DInitiati ves/96/07-3 1 .html 


(http://nces.ed.gov/ncesNe 

wPubs.html) 




http://www.ed.gov/Mailin 

gLists/EDInfo/msg00172. 

html 


http://www.ed.gov/NCbS/ 

ncesNewPubs.html 




http://www.ed.gov/Mailin 

gLists/EDInfo/msg00149. 

html 


http://www.ed.gov/NCbS/ 

ncesNewPubs.html 




http://www.ed.gov/Mailin 

gLists/EDInfo/msg00146. 

html 


http://www.ed.gov/NCbS/ 

ncesNewPubs.html 




http://www.ed.gov/Mailin 
gLists/EDInfo/msgOO 1 42. 
html 


http://www.ed.gov/NCbS/ 

ncesNewPubs.html 



O 
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Table IV-3: External errors to www.ed.gov 



Unavailable URL 


Link to URL 


Hyperlink Text 


1. http://www.itpolicy.gs 
a.gov/mks/yr2000/cio. 

htm 


http://www.ed.gov/offices 

/OCIO/year/ 


Y2K memoranda and 
progress reports 




http://www.ed.gov/offices 
/0CI0/year/ty2k 1 . html 


Y2K memoranda and 
progress reports 


2. http://www.ohioschool 
net.kl2.oh.us/404.asp 


http://www.ed.gov/Techn 

ology/sta_tech.html 


Ohio State Department of 
Education 


3. http://coe.ohio- 

state.edu/cete/ericacve 

/ 


http://www.ed.gov/bulleti 
n/fallw in 1 996/eric .html 


http://coe.ohio- 

state.edu/cete/ericacve/ 




http://www.ed.gov/offices 

/OVAE/daelnews/summer 

97.html 


ERIC Clearinghouse on 
Vocational and Adult 
Education 




http://www.ed.gov/offices 
/O V AE/daelne ws/ 1 0- 
96.html 


http://coe.ohio- 

state.edu/cete/ericacve/ 



To what degree do Department web pages meet effectiv e HTML coding practices? 

There are approximately 40 pages in which the Department is using incorrect 
height and width attributes of the HTML <img> tag. Although 40 pages within the 
extensive Department site may sound insignificant, the web industry standard is zero. 

Within HTML it is easy to falsify the height and width of images to make them 
appear smaller (or larger) to users. An image height and width may appear to be small 
within a web browser, yet the full image (in bytes) must be download before a user can 
view it in its false dimensions. Over a high-speed Internet connection, a user may not 
realize that the pages in Table IV-4 take a long time to download. Viewing the pages m 
Table IV-4 over a modem can take several minutes due to the immense images that web 

browsers download. 

To test the images, use Netscape Navigator to load any of the page locations listed 
in Table IV-4. When the page loads, right-click the image and select “View Image.” The 
View Image function will display the image in its true height and width, examples 
three, four, and five in Table IV-4 all three images are being loaded within the same 
HTML page {http://ww v.ed.gov/Technology/challenge/prior.htmr). The entire page size 
is over 640 Kb: ten times the size of an average web page. 
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Table IV-4: Large images with false height and width attributes 



Page Location 


Image Name 


HTML height 
and width 


Actual height 
and width 


Image 

Size 


1. http://www.ed.gov 
/offices/OPE/Direc 
tLoan/newsletter/j 
ul98/serving.html 


group.gif 


height=200 

width=290 


height=502 

width=732 


339 Kb 


2. http://www.ed.gov 
/offices/OPE/Direc 
tLoan/newsletter/j 

ul98/ 


diane.gif 


height=200 

width=290 


height=509 

width=741 


331 Kb 


3. http://www.ed.gov 
/Technology/challe 
nge/prior.html 


97cover.gif 


height=150 

width=200 


height=520 

width=681 


174 Kb 


4. http://www.ed.gov 
/Technology/challe 
nge/prior.html 


96cover.gif 


height=150 

width=200 


height-633 

width=629 


175 Kb 


5. http://www.ed.gov 
/Technology/challe 
nge/prior.html 


95cover.gif 


height=150 

width=200 


height=520 
width=8 1 1 


291 Kb 



To what degree do Denartment web pages f o llow effective design practices ! 

Pages with excessively large images 

The Department has several large images (measured in bytes) within its website^ 
web industry standard for average web page size, including graphics, is under 60 Kb (bee 
httn://www. inniitive.com/articles/design-guide.html where it is stated that users with 14. 
modems can receive no more than 45 Kb of data in 30 seconds). 

Table IV-5 presents example page locations where the Department has notably 
large images; each individual image exceeds 90 Kb. Table IV-5 provides the examples in 
decreasing order by file size. 
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Table IV-5: Pages with excessively large images 



fage Locuiiun 

1 http‘.//www.ed.gov/offices/OPE/Professio 
nals/S AR/sar 1 3 .html 


sarl3.gif 


121 Kb 


2 http 1 // w w w . ed. gov/ of fi ces/ OP E/Prof ess i o 
nals/SAR/sarS.html 


sar5.gif 


103 Kb 


3. httpV/www.ed.gov/pubs/SER/ProfEd/figu 

rej3 html 


figureJ3.gif 


102 Kb 


4. http://www.ed.gov/offices/OUS/eval/21ce 

nt/imDrove/sld030.html 


img030.gif 


99 Kb 


5 http://www.ed.gOV/pubs/FauliLine/covci .h 

tml 


cover.gif 


94 Kb 



The web page labeled number 1 in Table IV-5 shows a scanned ^ 

code” notice that is difficult to read, and exceeds web industry stands. Web pag 
number 2 in Table IV-5 shows a scanned “information review form. The form cou 

e^of n^vdue because it is difficult to read and prints illegibly. Web page 
disDlavs a hierarchical diagram for southern Maine. Once again it is clear that this is a 
Sn Id .mage which is excessively large and difficult to read. Web page 4 ,s part of a 
slide show. \he creator of this HTML page altered the height and wid* to ^ 
imaee At just under 100 Kb this image loads slowly over a modem. Web pap 5 is part 
“fdifferim slide show. At ftrst glance the 94 Kb image appe- 
dimension yet it downloads slowly. After importing this image into Adobe Phot^hop 
5 0 the study team found that the image had been saved as a maximum quality JPEG^ 
Maxtaum nuiT JPEGs are most often used for print media, and not for a web 
environment. As^a test, the study team saved the image as a medium quality progressive 
JPEG that reduced the image size to 35 Kb. 

Pages with excessive HTML text 

One hundred pages on the Department of Education website exc^d 70 Kb_ TOe 
averaee HTML text file size (excluding images) does not exceed 10 Kb. table tv o 
provides five examples of Department HTML text file tar beyond the 10 Kb average. 
Table IV-6 provides the examples in decreasing order by file size. 



Tablt IV-6: Pages with excessive hiiyll, lexi 
PqgeJ^ocation 


Page Size 


1 http://www.ed.gOv/offices/OSERS/RSA/RehabAct.htnil 


513 Kb 


2. http://www.ed.govAegislation/FedRegister/finrule/1997- 

1/021 197a.html 


412 Kb 


3. http://www.ed.govAegislation/FedRegister/proprule/1995- 

4/index.html 


361 Kb 


4 http://www.ed.gov/oftices/uHRI/BlueRibbonSchuul5/aic5.htnd 


171 Kb 


5. http://www.ed.gov/offices/OnA/Recognition/scholartable.html 


108 Kb 
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It is important to note that some of these represent pages where the content has 
been drawn verbatim from another source and thus may be difficult for the Departrnent to 
Ldify (for example, numbers 1 and 2 above come directly from the Federal Register). 

What naths do take through the first few layers of the website^ 

Path analysis provides a decidedly different performance measure than traditional 
counts of hits, page views, and visits. Whereas any page on the Department could be 
the most popular or most visited (through bookmarks, external search enpnes, etc.), path 
analysis Slows web administrators to see how users interact with the site. Path analysis 
of the Department site is important due to the number of pages the site contains 
Analvsis of a user’s path involves sequencing the document requests and an^yzmg the 
path created. “Path” is roughly synonymous with “page view,” the distinction here being 
that paths involve the further dimension of a chronological order. 

A “first request” is the page that a user initially visits upon connecting to the 
ed.gov domain. The most common first request is wvvw. ed.gov/mdex. 
Department homepage. Table IV-7 shows the top five first requests for September 20 - 

26, 1998. 



Table TV-7- Five most frequent first requests 

www.e d.gov/ 

/o ffices/OPE/Students/ 

/prog_info/SFA/StudentGuide/ 

/free/ 

/offices/OPE/express.html 

The top five first requests (Table IV-7) were then matched with their top ten 
“second requests.” For example. Table IV-8 shows the top ten second requests for 
/offices/OPE/express.html, which was the fifth most common first request. 



TqMp TV.R! First renuest with its most frequent second requests 



Firvt Request 



/ofFices/OPE/express.html 



.^prnnrl Reauesti^ 



^»ff]ppQ/OPF./5;tudents/ap r)lv/fext)ress.html 



/pnFDocs/QSfafsafmal /form.pdL 



/nffices/OPF/Students/ 



/PDFnocs/QSfafsafmal/ inst.pdf 



/nfFices/OPF /express.html 



/nffices/OPF-/Students/student.htm l 



/r> ffirp«;/nPF./Students/a nnlv/dwnload98-9,hmiL 



/nffices/OPEi 

/nrna info/S FA/StudentGuide/_ 



Site Server then reported the top ten “third requests” for each second request. 
Table IV-9 shows the top ten third requests for one example of a second request 

{/PDFDocs/98fafsafinaI/form.pdf). 



■Ip TV.Qi Fir^t and second requests witn ineir most iregucin . 

First Request 


/offices/OPE/expressJitml 




Second Request 


/PDFDocs/98fafsafmal/form.pdf 




Third Requests 




/PDFDocs/98fafsafinal/inst.pdf 


/PDFDocs/98fafsafinal/form.pdf 


/offices/OPE/Students/ 


/offices/OPE/Students/apply/fexpress.html 


/offices/OPE/express .html 


/offices/OPE/Students/apply.html 


/offices/OPE/Students/student.html 


/ 


/BASISDB/TITLE4/search/sf 


/DirectLoan/consolid.html 



Appendix IV-2 contains the complete path analysis that the research team compiled for 
the week September 20 - September 26, 1998. 



What areas of the homepag e do users most often select! 

The Department has over 40 links on its homepage, several of which are less 
freouentlv utilized than others are. Web industry standards state that more than eight 
navigational choices can confuse users. Choice and placement of links should be based 

on user path analysis. 

Table IV- 10 reveals where users are clicking once they have amved at the 
Depart-ment homepage. 



Table IV-IO: Top ten second requests from www.ed.gov 



xauAc Ji T A ^ 

S6C0nd R6QU6StS 


^ of aCC 6 SS 6 S 


1 


/ 


6,640 


2 


/stats.html 


941 


3 


/finaid.html 


798 


4 


/funding.html 


745 


5 


/databases/ERIC Digests/index/ 


611 


6 


/EdRes/ 


611 


7 


/Droexams.html 


605 


8 


/free/ 


557 


9 


/pubs/ 


550 


10 


1 /niihs/HDTnitiatives/98/98-09-1 1 .html 


534 


Total 


12,592 



(It should be noted that the top second request is the main page itself. This is a result of 
Lwsers requesting www.ed.gov. and the Department web server automatically 
redirecting users to www.6d.gov/ind6X.htntl.) 



When analyzing choices users make from the Department homepage, it is 
necessary to consider the physical layout of the homepage. Figure IV-2 shows the 
Department homepage and the four key clusters of links: 



1; Left-hand navigation bar 
2: Education Headlines links 
3; Most Requested Items links 
4: Horizontal navigation bar 



Figure IV-2: Department homepage four link areas 




Returning to the analysis of requests from the homepage (Table IV-10), certain 
trends emerge. Six of the top ten requests are located on the homepage left-hand 
navigation bar (1). One of the top requests is located in Education Headlines (2). Two of 
the top requests are located in Most Requested Items (3). None of the requests in Table 
IV- 10 are associated with the horizontal navigation bar (4). Please note that requests to 
search.ed.gov from the horizontal navigation bar could not be determined in this smdy. 
This type of analysis would require cross-referencing of log files. 

Analysis of the requests in Table IV-10 indicates that users are primarily relying 
on the vertical left-hand navigation bar (1 in Figure IV-2) to maneuver Aemselves 
through the Department website. The transaction logs show that most other links on the 
Department homepage are being used rarely. 

To what degree do some external search engines find and retri eve De partment we b 
pages? 

Teachers, students, and others seeking educational material may use external 
search engines to locate related websites. Department pages were generally 
underrepresented in the three external search engines tested. Teachers, students, and 
others seeking educational material currently can not easily discover the we^m ot 
Department resources available. Results of the external search engine trial (T^le IV- 11) 
imply that those looking for educational information may often bypass the Department 
website altogether. Considering the reliable, high-quality information available via the 
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Department website, it is unfortunate that the public does not have multiple access points 
to this information. 



Table lV-11: External search engine results 





Pathway Indexer T 


GOVBOT T 


AltaVista 


Query terms 


Number of ed.gov results (out of first 10) 

results 


- total number of 


National Center for 
Education Statistics 


0-4 


4-11685 


3-11303 


ERIC digests 


1-9 


10-5955 


6 - 3346 


Education headlines 


0-25 


0-5431 


0-2331010 


Education 


1-37 


0 - 3003 


0 - 24983357 


Department of 
Education 


0-25 


0 - 5957 


0- 199313 


Education reform 


0-25 


1 - 5962 


1 -27928 


Goals 2000 


0-27 


9-5878 


1 - 12608 


Adult education 


0-26 


0 - 5748 


0-88873 


Educational 

resources 


0-27 


0- 10 


0 - 82324 


Digest of Education 
Statistics 


0-1 


4 - 8874 


10-2310 



Do current nuerv forms support GET and PO ST query methods! 

GET and POST are the two methods used to query search engines. The GET 
method appends the search query to a browser’s URL, allowing users to bookmark their 
search results. The GET method has technical limitations placed on the allowable 
number of characters it can process. The POST request does not have any technical 
limitations on string length, but prevents users from bookmarking their search results. 

The study team intended to analyze the keywords that users searched for within 
the Department of Education search engine. After analyzing a few lines of the search 
engine log file, the study team realized that the Department uses POST rather than the 
GET method to query its search engine. Upon the study team’s request, Keith Stubbs, in 
an email message dated November 7, 1998, detailed the reasoning behind the Department 
choice of query method. Stubbs explained the “Cross-Site Index” service offered by the 
Department of Education, which allows users to search several databases with one query. 
The GET method (which is used by the majority of commercial search engines) has a 
query string length limit that would be exceeded by many “Cross-Site” queries. 

There are pages within the Department website that allow the user to query the 
internal search engine, but do not include the “Cross-Site Index” option (e.g., 
http://search. ed. gov ) . 
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The study team created an HTML form using the GET method (Appendix IV-3), 
but not allowing the “Cross-Site-Index” option. Using the new HTML form the team 
performed several queries, and successfully retrieved search results (Appendix T^). 
For comparison, the study team repeated the same test using an almost identical OTML 
form using the POST method (Appendix IV-5). Upon analysis, the study team achieved 
identical search results (Appendix IV-6). Ultimately the study team bookmarked and 
later returned to the search results created by the GET request. The study team could 
neither bookmark nor return to the search results created by the POST request. 

How can the Department increase the effectiveness of current log reports? 

WebTrends reports are being mn regularly within the Department of Education, 
however it does not appear that much action is being taken based on the results of these 
reports The study team found only eight unique Department IP addresses to the October 
WebTrends report between November 1 - 15, 1998. OESE reports were viewed only 
twice both external IP addresses. Such low usage of the October WebTrends report 
suggests that offices within the Department of Education may be overlooking the 
importance and usefulness of these reports. 



ISSUES AND OPTIONS 

The findings suggest a variety of options by which the Department can improve 
or enhance the www.ed.gov website. These options are presented below. 

Options 

How many broken links exist within the Department website! 

The Department keeps archives of numerous text documents dating back several 
years (e.g., http://www.ed.gov/PressReleasesf). Storing archives is certainly a laudable 
service for users, however it is also a common place to find broken links. The smdy team 
suggests using site management software (e.g.. Site Server) to help site admimstrators 
continue to find and fix these broken links. 

Users who follow internal broken links (Table IV- 1) within the Department 
website (or users who enter an incorrect URL) will receive the Department “Error 404 
HTML page (e.g., http://www.ed.gov/abcdefg). The Error 404 page is a single page 
within all websites that administrators should customize. 

Users who encounter the Error 404 page should not feel they have done anything 
wrong The traditional Error 404 page, which is large, bold black text, makes a user ee 
as if they have made a mistake (e.g., the message; “FUe Not Found”). Unless a i^er types 
in a URL incorrectly, the majority of 404 errors are the fault of the weosite, not the user. 
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The study team suggests that the Department consider redesigning its Emor 404 
„a.= ^rthe Depamint Error 404 page is cerrainly better rharr most on .he web 
isCm tor implement. The redesigned Error 404 page should look srmilar to all 
other pages within te Department site. Navigation should remain on the left, a bmner 
o^the tip and an explanation detailing the error that has oeeutted. High-quality Error 
404*page^s begin their explanation with the words “We’re sorry Two ° 

excelleffl Etijr 404 pages ate ZDNET (www.zdnet.com/abcdefg) and CMP 

(WWW. cmpnet. com/abcdefg) . 

To what degrp^ Ho Departme n t web pages meet effective HTML coding practice s? 

The Department has a vast number of web pages that contain Javascript and other 

advancS^mSgies. which the study team was happy to s^. 
archives pages dating back several years, which raises some concern. The rap d ch^ge 
in HTML and Web technology means that pages coded two years ago may no e p 
par today. While the study team found that most of the recent Department pages are 
properly coded, pages dating back to 1995 could use some revision. 

The study suggests that the Department put a team in place that is responsible for 
^dating Lb piges prior to 1997. This evaluation might include a review of the code for 
completion and adherence to standards for proper coding. 




Pages with excessively large images 



The Department scanned several forms and published them on its website. These 
forms are difficult to read, and large in bytes (e.g., http://www.ed.gov/ofte 
/OPE/Professionals/SAR/sar5.html). The study team recommends that the Dep^me 
recreate the scanned forms in HTML. The new forms will improve site design, and 
create increased interaction with the website. 

The Department has made efforts to improve forms on their website by 
them in Pottabfc Document Format (PDF). PDF furnishes a viable altemauve to 
fom, hut does not allow user Interaction with the website. The smdy team found several 
Seel wher^forms^^^ available in both PDF and HTML formats. The 
uses this practice inconsistendy throughout its site. The study te^ recommends the 
consistent implementadon for foims provided on the Department website. 

The Department also has scanned graphs and diagrams available “iSS 
http //www.ed gov/pubs/SER/ProfEdfigurej3.htm[).. The study team as 

te Sartment consider enhancing such images using a paphics progr^ such ^ 
Adobe Photoshop. The Department may reconsider publishing any image tha 
illegible after scanning and enhancement. 
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In addition to scanned images, Department web page developers 
exoloit the compression differences between GIF and JPEG image formats. The 
image formats offer substantial differences in terms of bit depth and qu ity^ 
discLsed in item five, Table IV-5, using the appropnate image compression 

drastically reduce image file size. 

Pages with excessive HTML text 

The Department website has HTML documents with large amounts of text (see 
Table IV-6). These documents, which in some cases exceed 100 pnnted pages, take a 
long time to download over low bandwidth media. The smdy te^ recoi^ends 
Upaitmem make long documents consistently avail^le tn ^ 

foimats In addition, indicative file size information should accompany lii^ to g 
"nts. Such practice better prepares users for the files they wish to download. 

As an alternative option, the Department may break up 
into several smaller HTML documents. For example, item four m Table IV-6 sh 
long list of schools listed alphabetically by territory. The 

several territories into one HTML document. In this example the ^e^ent ^nay 
choose to create separate HTML documents for each temtory, which will greatiy reduc 

file size. 

The Department might also review its current practice of providing copies of 
Federal Register files unedited. This may be difficult given the near-legal status of 
some of the documents however, their excessive size is problematic for users. 

What naths Ho users take through the firs t few layers of the websiM 

As explained earlier, path analysis can help web administrators sequence the 
documents users request and analyze the path created. The smdy te™ rccoi^ends 
the Department use software, such as Site Server, to create path anaiysis 
WebTrends can produce some reports, it appears to be unable to process ® ■ 

size generated by the site.) The Department may then analyze these reports to deterau 
popular and/or problematic user paths. The Department could also assess the ease with 

which users find particular items. 

What area.s of the homepa g e do users most often selec t2 

The Department should consider the critical importance of the ino^ co^on 

requests from ite homepage. This information may assist web ° 

3en7 alTd link posidoLg to reflect user needs. Web admimstrators should also 
examine the least popular links to determine if the links serve user needs or go 
Fo"ce, of this study suggests that users need finding lu^ to help them 

locate information on the site. However, findings from this secUon of the p 
that few users take advantage of the A-Z link. This suggests that users may overlook 
current finding aids available on the horizontal navigation bar. 
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The study team recommends the Department move finding aids to a more 
frequently accessed section of the homepage. For example, the Department ° 

create a “search button” within the vertical (left-hand) navigation bar. Theje^ch button 
could link to multiple finding aids, including the Department search page, A-Z , 
Cross-Site Search, programs and services listing, FAQ’s, and organizational ch . 
These redesign efforts should be followed by usability tests. 

Tn what degr>»f^ Hn external search eng i nes find and retrieve Department we ^ 

As findings show, the three external search engines tested do not frequently 
retrieve Deparanem web pages. Findings of the external search engine tesB 
from the lack of META tags in Department web pages. Search engines rely in p 
Xa^ METALS to build their index. The Department greatly increases the abtoy of 
Xh fnS to iLex its pages with the simple addition of carefully chosen HffiTA 
Xlhe study team recommends that the two types of META tags, “description and 
“keywords” would ideally be incorporated in all HTML pages. 

r>n rnrrent Querv forms support GET and POST query method^ 

Based on the findings above, the study team recommends that the department 

utilize both the GET and POST query methods. While the GET 

the POST query method, all other entries to the search engine should utilize the GET 
method. Using the GET method will allow web administrators to better analyze user 
search requests, and allow users to bookmark their search results. 

Hnw can th<^ Denartment increase the ef fectiveness of current log report^ 

As stated above, the Department currently publishes WebTrends reports on a 
monthlv basis Many large Websites run and analyze log reports on a daily basis due to 
r “mS'Xm at dam. The smdy team recommends the Department tun log 

reports on a weekly basis in order to ease analysis. 

Log analysis reports contain vast amounts of information, and can very easily be 
over 100 pages long. Experts in log analysis could hold traimng workshops to teach 
individual offices how to read and interpret a WebTrends report, and how to analj^e ^ 
act on the reported information. Furthermore, individuals responsible for mnmng repo^ 
should maintain regular communication (e.g., monthly reminder email messages) with 

offices regarding the reports. 
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Relational Database 



Although the Department currenUy uses WebTrends, it does not store log files 
in a relational database. Using a relational database (such as Microsoft SQL or Oracle) to 
loTSi ^ufd allow dte Department to generate sports showing lonpurin^ 

trends (e^g visits, paths through site, etc.). In addiuon, a relauonal database would ^low 

rDepS.;int » compare log flies from different servers, or compare log data to 
existing databases that the Department may have. 

Management Software 

The study team recommends that the Department invest in software simil^ to 
Microsoft Site Server due to its ability to handle the size of the log files and produce 
reports in addition to those produced by WebTrends. 



OESE FINDINGS 



Method 

The OESE log file analysis used the same methodology as described e^lier in the 
Sition ^ adLon. the study team performed “referral analysis” using WebTrench ^d 
Marketwave Hitlist software. The referral analysis shows external sites that link to th 

OESE website. 



Findings 



How many broken links exist w ithin the OESE website? 



Many examples of broken links exist within the OESE website, 
details examples of the three types of 404 errors that users may encounter, 
main report for further information about these errors. 



This section 
Refer to the 
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Table rV-12: Internal 


errors within www.ed.gov/of 


Ices/OESE/ 


Unavailable URL 


Links to URL 


Hyperlink Text 


1. http://www.ed.gOv/o 
ffices/OESE/news_a 
rchive/contact.htnil 


http://www.ed.gOv/offices/0 
ESE/news_archive/ 1-97 .html 


Contact us 


2. http://www.ed.gOv/o 
ffices/OESE/CEP/ha 
ppening.htinl 


http://www.ed.gOv/offices/0 

ESE/CEP/may22.htinl 


Happenings 




http://www.ed.gOv/offices/0 

ESE/CEP/mavl5.html 


Happenings 


3. http://www.ed.govAe 
gislation/GOALS20 
00/The 
Act/intro.html 


http://www.ed.gOv/offices/0 

ESEAeg_reg.html 


Goals 2000 



Table rv-13: Internal en 
Unavailable URL 


rors Wiuun um 

Links to URL 


Hyperlink Text 


1. http://www .ed.gov Aegi 
slation/GOALS2000/T 
he Act/intro.html 


http://www.ed. gov/ offices 
/OESEAeg_reg.html 


Goals 2000 


2. http://www.ed.gov/offi 
ces/legislation/FedReg 
is ter/announcements/ 1 
996-4/ 122796a.htmi 


http://www.ed.gov/offices 

/0ESE/news_archive/2- 

97.html 


Applications 


3. http://www.ed.govAegi 
slation/ESEA/disadvan 
/times495.htmi" 


http ://www.ed. gov/ offices 
/OESE/MEP/PrelimGuide 
^rochure.htIni 


Office of 
Elementary and 
Secondary 
Education 
/OESE) 
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Table TV-14; Exteroal errors from www.ed.gov 



Unavailable URL 


Link to URL 


Hyperlink Text 


1. http://www.neca.or 
g/funds 


http://www.ed.gov/office 

s/OESE/hot.html 


National Exchange 
Carrier Association 
has forms and 
instructions for 
applying for the e- 
rate. 


2. http://www.nwre!. 
or^national/region 
al-labs.html 


http ://ww w .ed.gov/ office 
s/OESE/MEP/PrelimGui 
de/appendix.html 


Regional Education 
Laboratories (Labs) 


3. http://www.enc.org 
/consortia.html 


http ://ww w .ed.gov/ office 
s/OESE/MEP/PrelimGui 
de/appendix.html 


Eisenhower 
Regional 
Mathematics and 
Science Education 
Consortia 



Altogether, there were over 20 unavailable resources within OESE. This number 
includes HTML pages and images. Appendix IV-7 is a complete listing of unavailable 

resources found within OESE. 



To what de>n-ee do Department web pages follow ef fective d esign practice s? 

Compression differences exist between GIF and JPEG image form^. (See 
earlier section for more information.) OESE web designers sometimes have used 
maximum quality GIFs (8 bit), when they could have used medium qu^ity JPEGs to 
reduce file size. One such example is an image named “mural.gif’ which is located on 
the Safe & Drug Free Schools Program web page 
(httD://www.ed.gov/offices/OESE/SDFSf). This image is nearly 58 Kb saved as an 8 bit 
gif, when saved as a medium quality progressive JPEG the file size can be reduced to 

Kb.’ 



Positioning of Pages, 

WebTrends reports show that the Safe & Drug Free Schools (SDFS) Progr^ web 
page has been the most popular page within OESE for the last three months. a is 
somewhat unusual is that this web page is more popular then the OESE i^elf 

Users may have a difficult time finding the SDFS program page from the OESt 
homepage. Users must click on three text links before arriving at this page. Reports show 
that most users going to the SDFS homepage find it through external se^ch engmes 
Sher websites, Ld lot through OESE. Please note that OESE is one of the few offices 
which use web page META tags. 
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After analyzing the WebTrends reports, study team found that the (SDFS) 
homepa^is also S nUer one page where users exit OESR The study 
that OESE did not design the SDFS page as an exit page. The investigators found that 
the overall size of the SDFS homepage is almost 70 Kb, which is not enoimous, but is 
pushing file size limits for modem users. This suggests that users cancel their requests 
due to Ae size of the file. An analysis of error logs (not undertaken in this report) could 

confirm this. 

Referral Analysis, 

Referrals (or links from other Websites) are the primary method for users to find a 
website Common referrals come from internal web pages, external se^h engine 
queries (e.g., http://www.altavista.com), or links from other websites. OE^ received 
over 20,000 referrals in the month of September. More than half of OESE referrals came 
from pages within the Department of Education website (e.g., 
http://wwwM.gov/gen_ed_org/prosram.html), while over 600 cme from Yahoo^ 
ih^://www.yahoo.com). Between the dates of September 20 and September 26, 1998 
OESE received 200 referrals from www.suck.com. 

Suck.com is published by Wired magazine and criticizes current events. On 
September 22, 1998, suck.com wrote an article on “The Safe f d^g-Free Schools ^d 
CoLnunities Act.” Within this article, suck.com links to the SDFS ^ 

publication generated a lot of traffic during that week. Appendix IV-8 provides 

complete copy of the article. 

How can OESE increase the effectiveness of current log re ports? 

OESE receives over 40,000 page requests a month, and 
spending over 16 minutes traversing the OESE website. Current OESE WebTren^ 
reports \http://www.ed.gov/intemaUwebstats) contain detailed information about the 
OESE user base that OESE staff could analyze on a monthly basis. 



OESE Recommendations 

How many broken links exist withi n the OESE website? 

OESE should consider running error-checking software (e.g.. Site Server) on a 
monthly basis. Running error-checking software will help find and elmunate any broken 
links (or broken images) within the OESE website. 

To what dein-fte do Department web p ages follow effective design practice s? 

OESE may want to reevaluate the navigation within its website, 
will help show the most popular pages within the OESE website (e.g., )■ 

may wit to promote programs such as SDFS by creating links in the top level (or second 
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level) of its website. OESE should also consider testing all web pages over a 33.6 
modem (the bandwidth of the average web user), to determine download speeds. 

Referral Analysis 

OESE can not prevent websites such as suck.com from linking to the site in a 
negative wav Regular analysis of the OESE log reports however, would inform it of 
sulh negative publicity. OESE would see. the referrals coming to the site and be able to 
trace the offending link. Had OESE known about suck.com and its negative referral, 
OESE could have written an article in response, defending the program and its work. 



CONCLUSION 

The changes in the web industry over the past year show 
management of websites requires not only the use of log analysis software, but also a 
complete web management application (including analysis software a 
The Department use of WebTrends indicates its desire to evaluate and improve its 
website m Department website transaction logs, present and future, contain only part 
of the information required to advance and improve online services. 

The suggestions and recommendations offered in this Section provide a beginning 
point for offices in the Department of Education to redesign and 
managing the website. Many of the recommendations and suggestions 
be coLiLred in light of recommendations offered in other sections of the report, e.g 
organization of staff to manage website development^ 

site as best it can given the limited resources that have been available for this • 
sl^of the problems indicated in this section are easily resolvable-given sufficient 

staffing. 

In addition, the Department may wish to consider impleinenting a number of 
training programs for webmasters to better understand and use the data produced by the 
wT i^agement reports. Without better understanding and use of these dam, 
webmasters are unlikely to rely on them as a basis for 
websites. Despite this concern, there is a good basis of '°S 

analyze the various websites and improve access to and use of the Department s websites 
Access to and use of the websites can be improved by further developing the log an y 
tools and techniques at the Department. 
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V. USER ASSESSMENT 



This section presents an overview of the user assessment component of 
project including the purpose and key findings of this portion of the study. The section 
continues by offering a detailed description of the data collection techniques, findings, 

and options for action. 



OVERVIEW AND KEY FINDINGS 

The user-based assessment represents one of the four assessment methodologies that 
the Syracuse team employed to assess the Department of Education website. The user- 
based evaluation emphasizes the raison d'etre of the website - the website users, and 
advocates site evaluation from the users' perspective. This assessment focused on die 
needs and perceptions of users interacting with the Department of Education website. 
The assessment had four main goals; 

• Investigate the perceptions of Department customer service personnel regarding 
the needs and types of users of Department information resources and usability 
problems with the website; 

• Investigate how a small sample of novice users utilize the site to find information; 

• Investigate how a small sample of expert users utilize the site to find information; 

• Investigate the degree to which a small sample of users could recognize and 
understand the patterns and groupings of links provided by each of the homepage 
help buttons; 

• Investigate how a small group of expert web designers interact with the Office of 
Elementary and Secondary Education site. 

These questions guided our analysis of data and led to the findings reported here. 



KEY FINDINGS 

Five themes emerged as key findings from the user assessment. These are: 

• The website www.ed.gov provides a wealth of useful education related 
information. 

The Department of Education provides a tremendous amount and wide variety of 
important information related to education, education funding, education research, 
educational program development and educational program funding. Users umformly 
praised the extent to which Departmental information was available on the site. 

• The website is currently structured to best support expert users rather than 
novices. 
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The Department of Education’s website provides excellent content. Expert users 
familiar with the Department’s structures and procedures can relatively easily use the 
website to satisfy information needs. The Department may wish to expand its user base 
to users a) who are less familiar with the Department’s structures and procedures and/or 
b) who have less experience using the web. If the Department aims to expand its website 
user base, it must adapt its website to meet the needs of less knowledgeable users. Such 
adaptations might include initiatives to reduce bureaucratic jargon and acronym use, 
improve the visibility of finding aids and provide brief descriptions, or discovery 
information” for all posted links. 

• Finding aids are currently not obviously placed for users. 

Due to the tremendous amount of information available at the site, the site's large size 
and complex structure, finding aids become key user resources. High quality finding aids 
need to placed in prominent and easy to find positions on the web site homepage and in 
subsite homepages. The Department has already developed high-quality finding aids 
such as the search features, A-Z index and various department and program listings. 
Non-expert users had difficulty finding these aids however. Improving their visibility is 
especially important if the Department chooses to expand its website user base. 

• Users have difficulty judging the relevancy or value of information that might be 
found via a particular link. 

Users sometimes have difficulty judging the relevancy or value of a particular link based 
solely on its title. Link descriptors, or “discovery information” aid users by allowing 
them to learn about the information a link provides prior to choosing that link. Many 
Department of Education webpages currently provide discovery information (e.g., the 
Publications and Products button page). Many other pages, however, do not provide 
discovery information. 

• The website currently relies too heavily on content drawn “as is” from print 
publications. The Department needs to determine to what extent it will support 
the evolution of the website to a user-oriented site. 

Faced with a great deal of pre-existing information to put up on the web, most 
organizations posted the information in its current form, and did not commit significant 
resources to considering more effective ways to present it. New information often exists 
first for print format and then is transferred “as is” to the web. 

The next stage for many sites is to evolve beyond this approach to a more user- 
oriented format. Initiatives for website evolvement might include projects to represent 
reports in multiple formats (e.g., MS Word file, PDF, html), to index reports and provide 
users links to indexed sections and to increase the amount of support material provided 
for novice users. It would also require a general resource commitment for innovation in 
information and service presentation in the web environment. 



DATA COLLECTION TECHNIQUES 



The project team utilized a multi-method data collection approach in the user 
assessment. A multi-method approach advocates collecting two or more sets of data on 
the same phenomenon using different data collection techniques. Analysis of the 
different data sets provides a richer explanation of the phenomenon of interest than an 
isolated data set can supply. The smdy team collected several sets of data regarding user 
experiences with the Department of Education website. The data collection acuvities 

included: 



• Interviews with Department of Education customer service personnel to 
investigate user types and their needs and common problems faced. 

• A task-based novice user evaluation of the website, and a task-based expert 
evaluation of the website, which measured users’ reactions to the site after using 
the site to answer a series of questions; 

• A user evaluation of the homepage buttons, in which users analyzed and reacted 
to printed out copies of the pages linked to by the left hand column home buttons 

on the www.ed.gov home page; 

• An expert web designer evaluation of the OESE website. 



The study team designed these activities to collect different types of data about user 
needs and user interaction with components of the website. The following subsections 
will provide details about each of these data collection activities. The second to the last 
section of this chapter will review the expert evaluation of the OESE site. 



Interviews with Department of Education Personnel 



The study team designed the first data collection activity to. 

• Identify customer service personnel’s perceptions of types of users of the 
Department’s information resources; 

• Identify customer service personnel’s perceptions of the information needs of the 
Department’s information resource users; 

• Identify key information resources used by the Department of Education customer 

service personnel; . 

• Document usability problems with the Department of Education website as 

perceived by customer service personnel; 

• Document customer service personnel’s perceptions of user expectations of e 
website. 



Customer service personnel within organizations develop an excellent and often 
underutilized understanding of users, user needs and user problems. Therefore, to collect 
data regarding users, needs and problems, the study team interviewed information 
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specialists and customer service experts at the Department. The Coordinating Team at the 
Department arranged interviews with staff with responsibilities associated with 
answering queries from the public. The study team asked to speak with personnel 
responsible for emsil, msil dnd phone inquiries. 

The study team spoke with thirteen employees onsite during the October 6-7, 1998 
field research session. The participants worked in a variety of offices and Principal 
Operating Components (POCs) and held a variety of different positions. The participants 
had an average tenure at the Department of Education of seven years. On average, they 
had worked providing information services for the public at the Department of Education 
for six years. They had worked developing internet-based services at the Department of 
Education for an average of two years. Participant Departmental affiliations included the 
National Library of Education, the Office of Intergovernmental and Interagency Affairs, 
the Applications Contractor, the Information Resource Center ( 1 - 800 -USA-LEARN), the 
Office of the Under Secretary/Office of the Deputy Secretary, the Student Financial 
Assistance Program, the Ed Pubs Service, and The Office of the Chief Information 
Officer’s Technology Center. 

Most participants worked an average of three to six hours a day interacting with the 
public, answering questions about the Department of Education. Most of the participants 
however, spent under one hour a day planning, implementing or maintaining internet 
based services. Participants had a great deal of experience with the Department of 
Education and the users of Department of Education information resources. Accordingly, 
they provided a wealth of information about the users of Department of Education 
information resources including users of the website, library and toll free help lines. 

Each interview lasted from forty minutes to one hour and focused on the Department 
of Education customer service personnel’s understanding of Department of Educadon 
users, user needs and user problems. Study team asked the employees open-ended 
questions, allowing the employees to explain what they felt was important regarding 
users and their needs. Additionally, we collected demographic information via a 
questionnaire. Data collection materials can be found in Appendix V-1. 



Task-Based User Assessment of the Site 

For the task-based user assessment, the study team created four tasks to guide the 
participants’ exploradon of the www.ed.gov site. Participants were allowed to proceed 
through the tasks at their own pace, until fifty minutes had past. At that point, the study 
team stopped all participants and had them complete a questionnaire on their perceptions 
and then participate in a group debriefing interview. Data collection matenals are 
available in Appendix V-2. 

The study team recruited two groups of users for the second data collection 
activity. The first group of four users, the “novice” group, included a teacher, a scl^l 
administrator, a district technology coordinator and an Education graduate student. The 
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second group of users, the expert group, consisted of five educational information 
specialists from the ERIC Clearinghouse. All participants had bachelor’s or master s 

degrees in Education. 

Group One 

The average age of the Group 1 participants was 30-40 years. The teachers had 
between 14 and 18 years experience in local public schools. The educational 
administrator had 8 years experience in administration at a local public elementary 
school. The technology coordinator had 8 years experience as a technology coordinator 
in a local rural public school district. 

Group one had moderate experience using the internet and little experience 
desi<^ning or evaluating web pages. The participants averaged four years of internet use. 
Two“ participants had never designed a web page. One participant had desired one, 
another (who worked as a district technology coordinator) had designed six. Group one 
rated its skill in navigating the WWW as moderate (3.5 with 1 as low and 5 as high), its 
experil^^^^^^ evalufting the design of websites as low to moderate (2.75) and its 
experience evaluating the content of websites as low to moderate (2.5). They rated their 
experience using the Department of Education website as low (2). 

Group 2 

Group two consisted of five educational information specialists from the ERIC 
Clearinghouse. Three of the five Group two participants had also worked as teachers, 
one for four years, one for six years and one for thirteen years. The remaining two 
participants had education degrees and had completed student teaching, but had not 
worked for a school district. The participants had worked as educational information 
specialists for an average of 2.3 years. 

Group two had higher levels of experience using the internet and designing and 
evaluating web pages. The participants had used the internet for an average of four years, 
and had designed an average of 12.4 web pages. The rated their skill in navigating the 
internet as very high (4.6 with 5 as high), and their experience with the Departrnent of 
Education website as moderate (3.4). They described their experiences evaluating 
website design as moderate (3.2) and their experience evaluating content as very high 

(4.6). 

Study team held separate data collection sessions for each group. The 
investigators asked the two groups of participants to attempt to answer four questions 
using the website as an information source. The study team chose four questions that 
would require the participants to navigate and explore several different parts of the 
website. The smdy team did not provide any instructions on how to find the appropriate 
information required to complete the tasks. Thus, each participant was free to explore me 
site in whatever manner he or she saw fit. The study team gave the participants i y 
minutes to complete the four tasks. 
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The study team performed a path analysis of the web server log files that captured 
the URLs that the participants used in attempting to answer the assigned questions. 
Appendix V-3 contains a copy of this log file. Log files record requests made to the 
www.ed.gov server for page downloads. Most PC web browsers however temporarily 
store, or cache, the images of recently visited pages. If a user visited a page and then 
returned to that page a few moments later, the web browser would pull the page image 
from its cache and not direct a request to the www.ed.gov. Therefore, the log file on the 
web server would not record a second visit to this page. In order to overcome this 
problem, the study team asked each participant to write down the title (not URL) of the 
pages they visited during their search. By comparing the participant lists with the log file 
of their visit, the study team could reasonably trace the participant's interactions with the 

website. 

After the participants completed the four tasks, the study team asked them to 
complete a short questionnaire to capture information about their impressions of the 
website and how using the website made them feel. The study team employed the 
WebMAC© website motivational instrument which measures the degree to which 
websites satisfy and motivate users (Small, R. 1997. Assessing the Motivational Quality 
of World Wide Websites. ERIC Clearinghouse on Information and Technology, 
ED407930). 

The Website Motivational Analysis Checklist (WebMAC©) assesses the 
motivational qualities of websites including appeal, usefulness and ease of use. 
WebMAC© collects data on four variables: 

• The degree to which website is engaging and stimulating. 

• The degree to which website contents are useful and credible. 

• The degree to which the website is organized and easy to use. 

• The degree to which users find the website satisfying and effective. 

Using the scores from these four variables, the checklist rates websites along two 
dimensions; 

• User’s expectation for success using the website. 

• User’s perception of the value of the website. 

Websites may fall into one of four quadrants created by the intersection of these two 
dimensions. Appendix V-4 illustrates the WebMAC© analysis scoring template. 
Websites should aim to score in the upper right hand quadrant of the scoring template, 
indicating high user expectations of success and high user perception of value. 

Finally, the study team coordinated a group discussion regarding the groups’ 
experiences with the website and their opinions regarding the purpose of the Department 
of Education website. 
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User Evaluation of Homepage Buttons 



The study team designed the third data collection activity to investigate the degree 
to which users could recognize and understand the patterns and groupings of links 
provided by each of the homepage help buttons, reflecting a concern expressed by the 
Department Coordinating Team regarding the utility and placement of the buttons. The 
team recruited a group of seven novice participants to evaluate the homepage buttons 
content. The group included two teachers, two school administrators, a district 
technology coordinator and two Education graduate students. 

The average age of the participants was 20 to 30 years, though three participants 
were older. The teachers had between 4 and 30 years experience in local public schools. 
The educational administrators had between 8 and 25 years experience in administration 
at a local public schools. The technology coordinator had 8 years expenence as a 
technology coordinator in a local rural public school district. All but one of the 
participants already had master’s degrees in education. The group had an average of 
three years experience using the internet, with two participants having little experience 
(zero and one year). Three of the participants had designed web pages. One participant 
had designed six, but the other two had created substantial numbers of pages (200 +). 
Group one rated its skill in navigating the WWW as moderate (3.3 with 1 as low and 5 as 
high), its experience in evaluating the design of websites as moderate (3.4) and its 
experience evaluating the content of websites as moderate (3.3). The two participants 
with less experience rated themselves substantially lower in these areas. The group as a 
whole rated their experience using the Department of Education website as low (2.4). 

For this data collection activity, the study team printed out the content of the 
pages represented by the eight homepage left margin buttons. The study team stnpped 
each of these pages of all its identifying and indexing information including titles, 
subtitles, URLs, navigation icons, and introductory paragraphs. The degree to which 
each page contained identifying and indexing information varied greatly with some pages 
containing little more than lists of unexplained links. The study team made copies of each 
of these pages and asked each of the participants to carefully read the contents of each 
page. The smdy team instructed the participants to then type a reaction to the page, 
including the following material; 

• A brief description of the page’s contents as the participant understood them. 

• A brief reaction to the page including all of the following; the participants ability 
to understand the contents of the page, the participant's emotional reaction to 
content on the page, any questions the participant had regarding the content of the 
page. The study team emphasized that participants could include both positive 
and negative reactions. 
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Finally, the study team asked the participants to physically edit each printed out 
page with a pen by; 

• Circling parts of the page which the participant had a difficult time understanding. 

• Crossing out any content that the participant considered inappropriate for the 
page. 

The team rotated the order in which participants evaluated each of the pages in order to 
ensure that each page received equivalent evaluatory effort. Thus, each of the button 
pages was either evaluated first or second by at least one evaluator. The data collection 
materials for this data collection activity can be found in Appendix V-5. 



FINDINGS 



Interviews with Department of Education Personnel 

The goal of this portion of data analysis was to identify major users of the site, the 
information needs which users attempt to fulfill at the site, users’ expectations of the site, 
and website based and non website based information which users find helpful in 
satisfying information needs. Analysis of data from the interviews with customer service 
personnel resulted in five major findings. 

• In general, the Department of Education has a well developed view of its user 
base and the needs of those users; 

• In general, customer service providers within the Department of Education seem 
satisfied with the content that the website provides, though they are not always 
satisfied with the arrangement of that content. 

• The growth in popularity of the web, combined with the varied nature of the 
Department of Education website audience, creates a situation in which the 
Department of Education is providing web-based services to people with little 
web experience. 

• Some users are confused about the roles and responsibilities of the Department of 
Education. These users expect the website to provide services it cannot. 

• Individual offices must construct and maintain their own FAQ’s and answer files 
which often duplicate information as no usable Department-wide networked 
resource exists. 

The remainder of this section provides more detail about the above findings. 

Types Of Users Of The Department Of Education Website 

The research team approached this question using the Department of Education 
website on-line evaluation report’s typology of users as a baseline of user groups. All 
participants agreed that the eight user groups identified by the on-line survey constituted 
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major user groups for the Department of Education website and other information 
services. These eight groups included: educational administrators, teachers and 
professors, study team and analysts, students, parents, librarians, writers and reporters, 
policy makers and legislators. 

In addition to the above listed user groups, participants noted that businesses and 
lawyers acted as information service customers. Businesses would contact the 
Department seeking information about how to market information to individual schools 
or school districts. In addition, businesses contacted the Department regarding marketing 
products directly to the Department of Education. Lawyers would contact the 
Department seeking statistical information. Additionally, various participants discussed a 
number of types of educational administrators (e.g. financial aid professionals, principals, 
state agency personnel), stressing the broadness of the current user categories. 

Finally, the department hasn’t addressed whether additional under-served 
audiences for' their materials exist. One participant suggested that the Hispanic 
population is better served now via Spanish speaking 1-800 call operators. The 
Department will need to decide how many resources to commit to including traditionally 
under-served populations with website materials. 

Information Needs Of The Department Of Education Informa tion Resource Users 

The smdy participants explained that users held a wide array of information 
needs, many outside of the jurisdiction of the Department of Education. For instance 
often times users, not understanding the role of the Department in local public educauon, 
would call with concerns or complaints related to their local or state educational services. 
Another common need involves rating or ranking local schools or school districts. 
Common information needs within the Department’s jurisdicdon included informaaon 
about publications, information about student financial aid, and grant eligibility 
information. 

Another noteworthy information need involves information about the Department 
of Education itself. Participants noted that internal and external users often needed 
information about the roles and responsibilities of the Department, the organizational 
structure of the Department, and contact information for specific programs and projects. 
One participant stressed the need for a more specific and detailed on-line organizational 

chart. 

Agency customer service personnel also mentioned the following as common 
information needs of web, mail and phone customers: 

• Parents (and students) complaining about school districts 

• Looking for publications 

• Trying to do research 

• Student aid questions and complaints (largely about use of online forms) 

• Availability of jobs/intemships at the Department 
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• Animal cruelty (how to teach about) 

• Statistical questions 

• Teacher certification questions 

• Information on whether college/programs were accredited 

• Which schools offer which programs 

• Information on default rates 

• Broken links 

• Title IV inquiries 

• What’s the school code (for financial aid) 

• Parents calling to find out how they can help their child learn 

• What parents can do if they think their civil rights are violated 

• What grants are available to groups, individuals 

These were needs remembered by customer service personnel and are probably a subset 
of the full range of questions/needs of users. 

Information Resources Used By The Department Of E ducation Customer Service 
Personnel 



Department customer service personnel reported using a variety of materials to 
satisfy users’ information needs. Participants regarded personal contacts with 
knowledgeable others in a variety of offices as a key information source. The importance 
of identifying, establishing and maintaining intra-organizational contacts emerged as a 
key theme of the interviews. In addition to personal contacts, customer service personnel 
described relationships with other customer services offices such as the Nationd 
Education Library, the Information Resource Center and the Office of the Secretary in 
each department. 

Customer service personnel made use of a number of Q&A, or frequently asked 
question files to answer users’ questions. The Information Resource Center (IRC), 
FREE, and the National Education Library made use of the IRC’s FOLIOview database 
of questions and answers. Library staff noted that network and hardware restrictions 
required them to dump a copy of FOLIOview into a desktop WordPerfect file for local 
Library use. Other participants had created their own personal Q&A files. No cross- 
agency help database currently exists. Respondents reported that they believed that other 
parts of the organization were asked similar questions and that sharing information on 
questions and answers to these cross-agency questions would be helpful. 

All participants described frequent use of the Department of Education website to 
satisfy users’ information needs. Participants noted that the web provided many 
advantages over local Q&A files including distributed maintenance responsibilities. A 
local Q&A file requires the local holder to periodically review and update information 
and phone numbers. With the Department website, the responsibility of updating 
information resides with individual offices. The “known usability problems” section of 
this report explains that unfortunately, some offices do not regularly update their website 
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files, resulting in the distribution of incorrect information about their programs and 
services. 

Many participants noted that the creation of the “ED Pubs” webpage on the 
Department site had vastly improved their ability to access and use information about 
Department publications. 

Known Usability Problems With The Department Of Educati on Website 

Study participants noted that their users had had a number of usability problems with the 
www.ed.gov website. These problems can be grouped into five major categories: 

• Confusion regarding Department services 

• Location of information 

• Incorrect or misleading information 

• Inexperienced users 

• Support for visual-auditory impaired guidelines 

Confusion Regarding RESPONsmiLinES 

Users come to the website with misperceptions regarding the roles and 
responsibilities of the Department of Education. These misperceptions lead them to look 
for information that does not exist. One participant suggested that the website include 
information about the roles of Department of Education closer to the website’s homepage 
in order to dispel many of the misunderstandings regarding the Department’s roles. 

Another responsibility issue revolved around the degree to which Department of 
Education employees should answer email correspondence faster than written 
correspondence. One participant noted that some Department employees treated email 
similarly to written correspondence, following lengthy “controlled correspondence 
procedures. These procedures significantly slow the rate of return correspondence. 
Email users however, expect more rapid responses given the ease of the media. To what 
extent should Department employees then answer emails faster than traditional paper 
based correspondence? 



Location Of Information 

Participants made a number of suggestions about the location of specific pieces of 
information to improve site usability. One participant suggested including information 
about the publishing department on the Publications and Products portion of the site so 
that users could contact that department more easily for additional information about the 
publication. Another suggestion was to increase accessibility to the Digest of Education 
Statistics by reducing the number of levels users had to navigate to locate it. (It is 
currently 2 clicks from the homepage which may suggest that the particip^t was 
misremembering.) Other users suggested making contact information more prominent on 
the Department homepage. Another suggestion included making the current 



organizational chart more detailed, as it does not provide information about the different 
offices within each department. 

Several participants noted that site redesigns often resulted in confused users as 
the location of specific pieces of information changed within the site. Although server 
side redirects will update bookmark users to the new location of their bookmarked page, 
site redesign compels users who enter the site via the homepage and navigate through to 
their needed information to relearn much of the site’s organization. 

Incorrect Or Misleading Information 

A number of suggestions revolved around the website’s inclusion of incorrect or 
misleading information due to certain offices’ infrequent webpage update practices. The 
participants noted that some offices did not remove time-sensitive information after its 
expiration date, leading to the confusion of some users. 

Another often noted problem revolved around the inclusion of “black hole” email 
addresses on office webpages. Black hole addresses consist of office customer service or 
help email addresses which no one monitors. Thus, users may send a message to these 
addresses and never receive any response. Some participants explained that they always 
worried about referring people to these office customer service or help email addresses 
for fear that no one would respond to the message. 

Inexperienced Users 

As the Department of Education moves to expand its user base to include more 
types of users, and as more of the general U.S. population experiments with internet 
usage, the problem of accommodating novice users will increase. Participants noted that 
novice users had difficulty with more advanced web-usage activities such as 
downloading and activating PDF files as well as with entering good search terms in the 
Department website search engine. For example, one participant noted that many novice 
users do not know to use quotation marks when entering related terms into the search 
engine, resulting in massive and useless lists of “relevant” search results. Participants 
suggested a number of items that would assist novice users including: non-case sensitive 
URLs, an FAQ button on homepage (which actually does exist), an “About Us” button 
and more information on meaning of choices on homepage. 

Support For Visual- Auditory Impaired Guidelines 

Although the Department of Education has prepared extensive visual-auditory 
impairment web design guidelines, not all offices that create webpages take the time to 
ensure their pages and services are compliant with these standards. Offices see the 
impairment guidelines as a last step prior to going live. In reality, consideration of the 
impairment guidelines need to come at the beginning of the design process in order to 
ensure that chosen applications and layout will support visual-auditory impairment needs. 




123 



III 



Other miscellaneous usability issues included the questionable comprehensibility 
of the Department’s home page buttons, the lag time needed to post speeches and 
information about current events, and the FAPSA forms lack of support for recently 
released versions of web browsers. 

User Expectations Of The Department Of Education Website 

According to the customer service representatives who participated in the user 
perspective portion of the study, many users expect that the Department of Education 
website will offer services outside of the Department’s jurisdiction. As mentioned in 
section a) of the known usability problems section, users believed that the Department 
would offer information about a variety of state and local education issues. 

The electronic nature of the medium seems to increase user expectations about the 
speed and timeliness of services. Some participants noted that email based users seemed 
to expect faster responses to their electronic inquiries than the typical response time for 
paper correspondence. Participants also noted that users expected the Department to post 
speeches and new publications within one day. 

Users also had high usability expectations for the site. They expected that the 
FAFSA forms would work with any browser. Through the interviews with the 
participants, it became apparent that users expected that email sent to any customer 
service email account would result in a response and appropriate action, even if it needed 
to be referred to another department. Thus, according to the participants, users did not 
always specifically direct their email help solicitations to the appropriate department. 
The participants also noted that users expected the Department of Education to satisfy 
their phone based help inquiries with minimal call transfers or dead end voice mail 
messages. 



Task-Based User Assessment of the Site 

The task-based assessment enabled participants to report their perceptions of the 
website based on actual usage of the site. The study team believed that the differences 
between the two groups (novice and expert) would result in the two groups having very 
different experiences and very different comments about the website. The investigators 
also believed that data from the two different user groups might allow the Department of 
Education to design pages to accommodate both low and high skill users. 

The study team analyzed performed three analyses associated with the task-based 
assessment including the identification of major focus group themes, statistical analysis 
of the WebMAC© questionnaire results and path analysis of users’ movements through 
the website. 
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Group 1 



Focus Group Results 



Data analysis of focus group discussion revealed the following seven themes: 

• Users’ inability to locate and use finding aids; 

• Misinterpretation of the "Programs and Services" button ; 

• Suggestions regarding the current layout of the homepage; 

• User frustration with browsing; 

• Inability of some users to navigate back to the homepage from deeper layers; 

• Inability of some users to interpret graphics as links; 

• Lack of confidence in using the search engine. 

The next section will provide more details about each of these themes and also discuss 
the participants’ views regarding the purpose of the www.ed.gov website. 

Inability to Locate and Use Finding Aids 

Participants complained about the difficulty of finding the finding aids (Search & 
A-Z) on the www.ed.gov site. Sezuching without finding aids required the users to 
browse through the site via seemingly relevant links, looking for clues. Most participants 
who found a finding aid found it on a page other than the homepage. 

Misinterpretation of the "Programs and Services" button 

All five participants unsuccessfully attempted to use the "Programs and Services'' 
button as a site index. They stated that they expected that the "Programs and Services" 
button would provide an easily scannable index of the programs and services offered by 
the Department (similar to the service provided currently by A-Z). They were 
disappointed and frustrated by the current page contents and did not consider it a 
satisfactory finding aid. 



Current Layout of Homepage 

Several participants complained about the current layout of the homepage. They 
felt that the current design first drew their attention to the "headlines" section - but that 
this section did not provide any finding aids to help them complete their tasks. Next their 
attention went to the left-hand buttons. They suggested that the Department change the 
color of the buttons to make them "stand out more". As noted above however, the 
buttons also did not provide satisfactory finding aids. Most of the participants never saw 
the search/a-z/faq/directories tool bar at the top of the page. (This parallels the results of 
the transaction log analysis — see Section IV of the report.) 
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User Frustration with Browsing 



Because the users had to rely on browsing to find relevant material, they became 
very frustrated with the amount of irrelevant material they had to browse through. As 
one participant explained, "I hate surfing". Participants wanted a simpler way to quickly 
focus on topics of interest within the site. 

Navigation Problems 

The participants had problems returning to the Department homepage during their 
searches because many office and program pages did not provide return links to the 
Department homepage. Also, the varying layouts of the office and program homepages 
created some frustration and created disorientation and feelings of "being lost". 

Graphics Problems 

Some participants had trouble distinguishing hypertext links. Either they found 
the links too lightly colored to distinguish, or they had difficulty distinguishing between 
plain graphics and image-map links. "I can't tell what is a button!" one participant 

exclaimed. 



Search Engine 

The two participants who used the site's search engine expressed a lack of 
confidence in the engine. They were confused as to the number of different search 
engines at the site because the search feature was marked differently on different pages. 
The participants also felt that the search engine required some searching/query 
developing expertise for good use. One expressed annoyance that the title lists of key 
word search results often did not include the key word. Thus the participant could not 
judge how the relevancy of the use of the keyword in that document. Another search 
function user expressed frustration with her inability to do more sophisticated searching 
without reading the search engine directions. 

Purpose of Website 

The study team also asked the participants to discuss the purpose of the 
www.ed.gov website. The participants all agreed that the site should provide access to 
information, but they disagreed about the primary audience. One participant thought the 
site should show school accountability for parents and taxpayers. Another thought it 
should highlight and describe services for educators - a "what can you do for me? 
approach. One thought it should provide classroom materials. Another disagreed and 
thought other sites like ERIC offered those services and that ed.gov should concentrate 
on administrators. Another suggested that the site tie together all governmental education 
institutions including state and local level institutions. Finally, one noted that she felt the 
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purpose of the website was to reduce the number of phone calls the Department had to 
take. 

Path Analysis 

Analysis of user log files combined with data from the participants notes 
regarding the pages they visited confirmed and added information to several of the 
findings from the findings from the focus group. Path analysis showed that four of the 
Group 1 users used the A-Z finding aid, and that two of the Group 1 users used the search 
function. Furthermore, one participant used the AltaVista commercial search engine. The 
path analysis also shows however, that all of the participants did not find or use the 
finding aids until well into their search sessions and often on pages other than the main 
home page. This confirms the focus group finding that Group 1, novice participants, had 
difficulty locating the finding aids within the site. See Appendix V-6 for a copy of the 
users’ paths. 

WebMAC© Results 

As discussed earlier, the Website Motivational Analysis Checklist (WebMAC©) 
assesses the motivational qualities of websites including appeal, usefulness and ease of 
use. Group one averages for each of the four variables are listed below. The highest 
possible score is a 30. Small and Amone indicate that a score under 20 indicates a need 
for improvement. 

Degree to which website is engaging and stimulating (ES) 15.4 (with 

outlier removed -18) 

Degree to which website contents are useful and credible (UC) 19.4 

Degree to which the website is organized and easy to use (OE) 12 

Degree to which users find the website satisfying and effective (SE) 16 



WebMAC© combines the ES and UC scores to create a score for the user’s expectadon 
for success using the website. Group one scored 34.8 points on this dimension. 
WebMAC© combines the OE and the SE scores to create a score to represent the user’s 
perception of the value of the website. Group one gave the Department website a score 
of 28 on this dimension. 

Websites may fall into one of four quadrants created by the intersection of these two 
dimensions. Appendix V-7 illustrates the results on the WebMAC© analysis scoring 
template. 
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Group Two 



Focus Group Results 



The second group was composed of information professionals familiar with 
education material. The study team expected that the second group would use more 
sophisticated search techniques and finish the assigned tasks much faster than the first 
group of teachers and administrators. 



This expectation was only partially fulfilled. The second group did use much 
more sophisticated search techniques. All of the participants used the search feature m 
several used the cross-site index within the search feature. One participant used the A-Z, 
index and three participants used the Programs and Services button. Most of the 
participants however did not finish the tasks any faster than the less experienced Group 1 
participants. 



Data analysis of the Group Two focus group discussion revealed the following seven 
themes: 



• Users complained about the lack of 'discovery information' or link descriptions; 

• Users complained of the overuse of bureaucratic language; 

• Users complained of the lack of easy, intuitive interfaces; 

• Some users experienced navigation problems; 

• Suggestions regarding the current home page design. 

Lack of Discovery Information 

Several of the participants expressed dissatisfaction with the "discovery 
information,” or text which describes the content of a given link. In general they felt that 
the text surrounding links did not provide adequate cues regarding the content of that 
link. One participant claimed that the summaries given for a link did not give enough to 
evaluate the relevancy of links. Another explained that you can't decide by looking at a 
link if it will lead to your desired information. Another asserted during a general 
discussion that much of the discovery information was misleading either because the 
information may not have accurately reflected the content to which the link connected or 
that the discovery information may have used language which she did not perceive as 
accurately descriptive of that content. 

U SE OF Bureaucratic Language 

Several users also complained about the overuse of bureaucratic language at the 
site. They felt that discovery information and link titles used language more appropriate 
for Department of Education employees than average users. The researcher asked the 
participants who had used the Programs and Services button to find information if they 
thought less experienced users would also be able to use Programs and Services to find 
information. They felt that more novice users would find Programs and Services 



confusing. They felt that one had to have a good understanding of the Department 
structure, and what things qualified as "programs" or "services" to find information 
through that button. They also felt that the website did not use consistent headers. 

Lack of Easy, Intuttive Interfaces 

Participants complained about the non-intuitive arrangement of the site. One 
participant explained that she always uses the cross-site index when searching for 
information at the Department of Education website because otherwise she would never 
be able to find anything. One participant suggested that the Department should construct 
separate indexes for teachers, parents and other user groups that would provide a more 
inmitive interface to department resources. Several users asked about the "Parent’s 
Guide to the Department of Education" which they remembered as a good idea which 
was poorly implemented. 

Navigation Problems 

Similar to Group One, participants in Group Two also complained about 
disorientation at lower levels resulting from lack of navigational tools and unmarked site- 
exiting links. Several participants complained that lower level pages did not provide links 
back up to higher level pages or to the Department home page and did not include 
navigational buttons. Also, users could often not determine if they had left the 
Department home page or not. The page did not provide clear warnings when directing 
users to offsite pages. 



Home Page Design Suggestions 

Participants also complained about the "clutter" of the homepage. They suggested 
that the Department not list all the headlines on the front page, but on a second layer page 
in order to free up space on the homepage. 

Purpose of the Department of Education Website 

Participants in Group Two also did not agree about the purpose of the Department 
website. Some felt it should serve all the needs of many groups including parents, 
teachers and smdents. Others felt it should not try and serve all needs and should 
concentrate on certain issues such as legislation and their programs. Several participants 
suggested that the Department not try to provide all education related information, but 
rather concentrate on acting as a pointer to other sites which provide information. Other 
participants weren’t sure what information the site should provide, but just stressed that 
the Department currently provided too much "who we are" information. 

Path Analysis 



In contrast to the novice users, the expert users located and utilized the site 
finding aids more quickly. Four out of five of the expert users used the site search 
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function. The one user who did not use the search feature told the study team that she 
normally did use the search function, but chose not to use it in this instance in order to 
see if she could find the information without using it. Most of the expert users began 
utilizing the search feature earlier in their site interactions. 

WebMAC© Results 



Group two averages for each of the four variables are listed below. As mentioned 
earlier, the highest possible score is a 30. Small indicates that a score under 20 indicates 
a need for improvement. 



Degree to which website is engaging and stimulating (ES) 

Degree to which website contents are useful and credible (UC) 
Degree to which the website is organized and easy to use (OE) 
Degree to which users find the website satisfying and effective (SE) 



15.75 

16.63 

16 

18 



WebMAC© combines the ES and UC scores to create a score for the user’s expectation 
for success using the website. Group two scored 32.38 points on this dimension. 
WebMAC©combines the OE and the SE scores to create a score to represent the user’s 
perception of the value of the website. Group two gave the Department website a score 
of 34 on this dimension. Appendix V-8 illustrates the WebMAC© results plot for the 
expert group. 



Comparison of Group One and Group Two Scores 

Group one and group two both gave the website around 15 or 16 points to indicate 
the degree to which website is engaging and stimulating. The more novice users in group 
one scored the website slightly higher on the variable measuring the degree to which 
website contents are useful and credible. Interestingly, the novice users gave the website 
a significantly lower score for the organized and easy to use variable than the expert users 
(12 vs. 16). Appendix V-9 illustrates the WebMAC© results plots comparing both 
groups. 



Evaluation of Homepage Buttons Content 

This section reviews the findings of the button pages content evaluation. It first 
outlines the major themes expressed in the post evaluation focus group. It then provides 
an overview of participants' comments about the button pages, arranged by page. For 
transcripts of participant reactions to each page, see Appendix V-10. 

Focus Group Themes 

In the post-session focus group, the participants' comments centered around the 
following two themes: the need for the pages to be “eye easy” and uncertainty about the 
purpose of certain links on some pages. 
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Need for make pages “eye easy” 



The participants had higher opinions of those pages that they found easier to look at. 

Pages that were easy to look at contained several of the following characteristics: 

• Discovery information for each link; 

• Extensive use of bullets or numbering. In some cases, the participants had a difficult 
time distinguishing the end of one bunch of grouped material from the beginning of 
another. For example, on the Presidents and Secretary’s Initiatives page, the 
participants could not determine if links belonged to the paragraph located directly 
above or below the links. They suggested numbering the points and increasing 
spacing between the points/related links and the next point. 

• More extensive use of subheaders. Because the study team took out all the 
subheaders which the pages provided, participants naturally complained about lack of 
subheaders. However, participants complained about lack of subheaders on all pages, 
even those pages that did not originally include subheaders (e.g. Programs and 
Services). 

Also, participants expressed a particular wish that some pages with particul^ly 
extensive link lists would not list out all of the links on the 2"** level, but rather just 
provide a general category link to a 3"* level which would then list out all links. For 
example, on the “Other Sites” page, the participants suggested making a “state 
resources” index link which one could choose and then see the more extensive list of 
state resources. 

• Links presented at the beginning of discovery information (not in the middle or end). 
For instance, the participants did not like the Direct Loan Web Site link on the 
financial aid page because of the links location at the end of the discovery 
information. 

• Standardized font sizes. For example, the participants found the mixed font sizes on 
the Presidents and Secretary’s Initiatives page distracting. 

• More explicit point/subpoint strucmre. Some pages use large bullets for subpoints 
and small bullets for main points (e.g. Statistics Page). The participants found the 
lack of coordination between bullet size and organizational structure confusing. 

The participants did not understand the purpose of some links on certain pages. 

Given their understanding of the content and purposes of the page, the 
participants did not understand the location of certain links on certain pages. See the 
following individual page analysis for examples. 
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Participants' Reactions by Page 



This section summarizes the participants' reactions by button page. Appendix V- 
10 provides a full transcript of each user’s reactions to each page. The section then 
provides a review of the options that the Department of Education might consider when 
making use of this data. 

Ed Offices and Budget (www.ed.gov/about.html) 



The majority of participants identified this page as an introductory page to the 
Department. They felt that including descriptors or discovery information for each link 
would make it easier to judge the relevancy of links to information needs. Some also felt 
the links might appear in a more useful order. 

Other Sites (www.ed.gov/EdRes/index.html) 

Participants had a difficult time understanding the theme or purpose of this page. 
Some identified it as a “educational resource organizational directory” but others becarne 
confused by the variety of information provided. The participants who looked at this 
page seemed overwhelmed by the number of links presented. They suggested organizing 
links by themes and providing indexes or headers for those themes. Also, the participants 
pointed out the need for discovery information about each link. They had a difficult time 
judging the relevancy of each link based on just its title. 

Research and Statistics (www.ed.govstats.html) 

Some users were able to identify this page as a list of links related to educational 
research. The participants noted that this page also did not provide discovery information 
for links. They suggested that better organization of related links into groupings with 
headers might ease page browsing. 

Programs and Services (www.ed.gov/programs.html) 

The participants were not able to identify the purpose of this page beyond an 
index page of some sort.” However, they appreciated the discovery information that this 
page provided for all its links. Further, they appreciated that the page explained all 
acronyms. They suggested that designers might improve the page by grouping like links 
together and proving headers. 

Student Financial Assistance (www.ed.gov/finaid.html) 

The participants all identified this page as a page for financial aid resources. 
They appreciated the discovery information provided for most of the links on the 
Financial Aid page. They noted however, that this page did not fully explain all 
acronyms and did not always place links at the beginning of the discovery information - 
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forcing the user to scan the whole paragraph. They also suggested that this page attempt 
to group like links and provide subheadings 

Funding Opportunities (www.ed.gov/funding.html) 

Participants were able to identify this page as a source of information regarding 
federal funding for education programs. They suggested adding discovery information to 
all the links. Without the discovery information, some links did not seem relevant (e.g. 
Safe and Drug Free Schools). 

President’s and Secretary’s Priorities (www.ed.gov/inits.html) 

Most of the participants identified this page as an index page of information 
related to educational initiatives. They found the order of presentation of the information 
on the page confusing however. They suggested moving the “gist of the page” or the 
Secretary’s introduction, from the middle to the top. They also did not like that the page 
presented the points out of order. They did like the presentation of related links under 
each point. They suggested standardizing font sizes in order to improve the appearance 
of the page. 



News and Events (www.ed.gov/news.html) 

Participants were able to identify this page as a press release and publications 
page. They found the amount of information and lack of indexing overwhelming 
however. They stressed the importance of grouping like links and providing 
indexing/headers to improve usability. They also suggested proving discovery 
information for each link so that readers could more easily judge relevancy. Finally, 
several participants questioned the relationship of information at the end of the page to 
the page’s main theme. 

Publications and Products (www.ed.gov/pubs/index.html) 

The participants identified this page as a publications and information resource 
directory page. Several felt that this page was similar to the Programs and Services page. 
Many felt that this was the best page. It provided good discovery information for each 
link which gave the participants confidence that they could choose the most appropriate 
link for their needs. They suggested improving the page by grouping like links into 
categories and providing indexing for those categories. 



ASSESSMENT OF OESE WEBSITE 

This section will provide an overview of the research method used by the 
investigators to evaluate the Office of Elementary and Secondary Education's (OESE) 
website under the www.ed.gov domain. This section will also review the findings of that 
assessment and make review various design options that OESE may consider. 
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Method 



The study team employed the "expert evaluator" technique in order to evaluate the 
OESE website. The expert evaluator method centers around using usability experts to 
identify usability problems in systems design. It is important to point out that these 
experts are not able to comment on whether the site meets organizational objectives. 
Answers to questions, such as, “Does the site’s content represent the content available 
from the organization?” or “Does the site reach the appropriate audiences?” are not 
possible. 

Evaluators 



The study team recmited four highly experienced website designers to complete 
the expert evaluation of the OESE website. The web designers had considerable 
experience using the WWW (average 5 years), and high levels of experience designing, 
evaluating design and evaluating content within websites. The web designers had each 
created between 50 and 200 plus websites each over the last several years. The web 
designers' high levels of expertise qualified them to serve as expert evaluators of the 
OESE website. 

Sample 



The OESE website consists of eight major subsites and a series of introductory 
pages. To evaluate the site, expert evaluators examined the 1'* and parts of the 2 level 
of each of the eight subsites, in addition to browsing through the other shallower links 
included off the main homepage. In pursuit of this aim, the researcher instructed the 
evaluators to browse at least one link down in each of the eight subsites. This technique 
guaranteed that evaluators would evaluate much of the same material (the f level of the 
subsites) while also guaranteeing that at least one evaluator would look at several 
different 2"“* layer pages within each of the subsites. 

Subsites Examined: Compensatory Education (CEP) 

CSRD 
Goals 2000 
Impact Aid 

Office of Indian Education 

Office of Migrant Education 

Safe and Dmg Free Schools 

School Improvement 

Main homepage and Associated Links 

In order to ensure that the web design experts would pay equal attention to each 
subsite, the study team rotated the order in which the experts evaluated the sites. Thus, 
one evaluator began by evaluating the CEP site; another evaluator began with the Impact 
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Aid site, and so on. This rotation method guaranteed that at least one evaluator would 
evaluate every subsite, maximizing evaluation coverage, and that each subsite would be 
evaluated at the beginning of the evaluation period before the evaluators became tired. 



The study team gave each evaluator a set of instructions with suggested criteria 
for website evaluation. Please see Appendix V-11 for instructions. The instructions 
suggested that the evaluators evaluate each of the subsections, and the OESE site as a 
whole, on the following five criteria: 



Orientation 

Design 

Navigation 



Quality 

Customer Service 



Overview, scope, mission, liability, copyright if appropriate 
Aesthetics, consistency, and appropriateness 

Minimal user skills required, clear indicators/links, link provided 
to appropriate homepages (OESE, ED), obvious and easy to 
understand finding aids 

No dead end links, outdated information, incomplete pages, 
appropriateness of content 

Obvious and easy mechanisms to provide feedback, obvious and 
easy to understand help features 



The study team presented the evaluators with the above five criteria as guides. They also 
stressed that the evaluators could include comments on any additional material they felt 
important. 

The evaluators examined the OESE site simultaneously for one hour, using 
equivalent desktop computers and Internet connections. The study team cued the 
evaluators every ten minutes in order to encourage them to continue moving through the 
site and evaluate as many subsites as possible. After the one hour evaluation period, the 
study team coordinated a group discussion in which the evaluators shared their 
experiences and their opinions of the OESE site in general and the specific subsites that 
they evaluated. After a thirty minute discussion, the study team ended the data collection 
event and thanked the participants for their time. 



Findings 

This subsection reviews the findings of the expert evaluation. First it outlines 
some of the general reactions of the evaluators which apply to the entire OESE site. 
Second, it reviews the evaluator’s comments for each of the subsections within the OESE 
site. The report continues with the final section, which outlines some of the options for 
the OESE site designers based on the results of the site evaluation. 

Focus Group Themes 

Because many of the evaluators examined different parts of the OESE site, many of 
their comments apply to only certain parts of the OESE site. Certain themes, however, 
emerged from the post-evaluation group discussion which apply to the entire site. 
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Target AuDffiNCE 



The four evaluators felt that the OESE site primarily catered to users with high levels 
of knowledge about the website and OESE programs and agency officials equally 
familiar with OESE and its programs. For instance, the evaluators noted that the site 
assumed a given level of knowledge regarding grant writing. The site presented helpful 
information about grants, but very little information about how to write grants. Thus the 
site assumes that the people interested in grant writing information have experience 
writing grants. The evaluators felt strongly that OESE should also try to include 
information which would provide more of a step-by-step approach for grant writing in 
order to assist less experienced users. 

The evaluators also noted that the order of links on each of the programs' introductory 
pages (e.g. placing the Director link first), did not cater to the needs of non-expert users. 
Finally, the evaluators noted that the site tended to use terms that assumed a certain level 
of knowledge regarding agency functions (e.g. 'advance organizer, abstract'). The 
evaluators suggested that OESE make efforts to simplify language on the site as much as 
possible. 



Purpose of Site 

The evaluators felt the site needed to transition from the current "brochure-ware" 
orientation to a help orientation. They explained that brochure-ware sites provide 
information, but little assistance. Additionally, brochure-ware sites tend to provide fairly 
shallow content. They elaborated that the current site fulfilled its brochure-ware/public 
record obligations well, but needed to move on and experiment with different help- 
oriented features such as the grant-writing help, or more extensive FAQ listings. 

Inconsistency in Design 

Evaluators approved of the simple and clean designs used on the vast majority of the 
OESE pages. They found a great deal of inconsistency however, between OESE 
subsections and suggested that this inconsistency might confuse users. 

Furthermore, the evaluators approved of the introductory page developed for each 
subsection, but felt it often conflicted with or overlapped with the 'About Us' link present 
on many subsection pages. They agreed that the introductory page clarified the purpose 
and mission of many of the subsection sites. They made several suggestions however, for 
further improving the site by maximizing the usefulness of this page. They suggested 
using this page as an index to the subsection. They also suggested making the icons 
along the side of the introductory page more noticeable and more consistent with the text 
based links. Finally, they questioned the need to list the Director link first on the 
introductory page. 
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The evaluators also noticed inconsistency between subsections in their ability to 
provide user feedback or contact OESE officials with questions. Some subsections 
provided clear and easy feedback options. Others provided feedback options, but buried 
it within the 'About Us' page. Finally, other subsections did not provide feedback options 
at all. 



Sue Exit POINTS 

The evaluators pointed out that most subsites gave users no warning when 
proving links to non www.ed.gov sites. 

Report Formats 

The evaluators suggested that the OESE staff attempt to move one step beyond 
posting reports and materials on line and begin html indexing these reports. HTML 
indexing would allow users to skip immediately to sections of interest without scrolling 
through an entire document. Furthermore, html indexing would allow users to print out 
only relevant portions of a document instead of printing out the complete document. 
Finally, the evaluators suggested that OESE attempt to increase the number of formats 
for downloaded material. Currently OESE relies predominantly on PDF. The evaluators 
suggested the goal of making documentation available three formats: PDF, html-indexed 
and MSWord format. 



Dated Material 

The evaluators noted that OESE often dated their on-line material. They 
suggested OESE attempt to improve by also including a data or report 'expiration date'. 
The expiration date would indicating how long data/material is good for, allowing users 
to know how long they could reasonably use this data in a report, and when the next 
report will become available. 



Searching Ability 

The evaluators could not find an easily accessible searching mechanisms within 
OESE site. 

In summary, the evaluators felt the OESE site had strengths: no over-design or 
overuse of graphics or animation, and use of concise introductory pages to each 
subsection. They felt however, that the OESE site needed to evolve to the next level of 
website design. The site needed to move from 'brochure-ware' to a help orientation. 

Many of the themes presented in this subsection apply to all the pages within the 
OESE site. Appendix V-12 provides verbatim comments from participants for pages 
examined. 
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Options for OESE website 

Given limited time and resources, the Office of Elementary and Secondary Education has 
at least three options; 

1. Maintain existing website design. OESE could choose to focus resources 
on programs and services other than the OESE website. 

2. Expand the scope of on-line information. Focus energy on getting things 
on-line that currently are not on-line, or only exist on-line in the tree 
format (e.g. Impact /^d. Office of Indian Education). 

3. Improve the usability of currently existing web pages. OESE could 
choose to focus its energies on evolving its current pages to a new level of 
usability through page redesign and usability testing. 

If OESE chooses option number three, it may consider any of the following general 
usability oriented projects suggested by the expert evaluation; 

• Clearly identify target audiences, make certain design offers options catering to 
multiple target audience groups; 

• Increase usability for non-experts through increased use of plain language, more 
help-oriented features (FAQs, grant writing help), more indexing of long lists of 
links (Goals 2000 and CSRD); 

• Increase consistency of design across pages, including clear and easy navigation 
features; 

• Improve visibility and robustness of user feedback features; 

• Resolve the introductory page/about us conflict. Maximize utility of introductory 
page with overviews and indexes; 

• Clarify exit points; 

• Expand the number of available download formats; 

• Expiration date all materials; 

• Improve searching capabilities (search feature, A-Z index link); 

• Highlight finding aids on OESE homepage (search, A-Z index link. Programs and 
Services listings). 

Implementing any and all of these recommendations has the potential to improve ±e 
usability of the OESE website. 



OPTIONS FOR WWW.ED.GOV 

The study team congratulates the Department on its website. The www.ed.go^^ site 
currently serves a large audience and serves it well. The site presents an extensive array 
of resources to users. These resources are buttre;;sed with several finding aids. As ±e 
Department looks to extend its website, it can do so in a number of ways; by extending 
the site’s reach to new audiences, by extending its functionality, and by including new 



finding tools. The usability assessment points to some areas in which the site can be 
improved as it continues to grow. 

This section presents a series of options that the Department of Education may choose 
in order to make use of the findings presented in the user-based portion of this study. The 
section outlines general website usability design approaches which apply to all pages 
instead of focusing on design issues only applicable to individual pages. The options 
presented in this section relate directly to the key findings presented at the beginning of 
Section HI. The user evaluation presented five main findings: 

• The website www.ed.gov provides a wealth of useful education related 
information. 

• The website is currently structured to best support expert users rather than 
novices. 

• Finding aids are currently not obviously placed for users. 

• Users have difficulty judging the relevancy or value of information that might be 
found via a particular link. 

• The website is at the stage of development in which previously existing print 
content forms the basis for much of the web content. TTie Department needs to 
determine to what extent it will the evolution of the site to a true user-oriented 
site. 

The options center around the degree to which the Department wishes to support a 
website evolution. The Department might choose one of the following options; 

• No/Low commitment 

• Medium commitment 

• High commitment 



No/Low commitment 

In this scenario, the Department of Education decides not to commit significant 
resources to manage an organized evolution of the website. Individual POCs may 
commit resources to website evolvement, leading to a Department of Education website 
with highly varying levels of webpage sophistication. The Department still maintains 
and provides high quality on line educational information, but does not commit resources 
to improving access to that information for non-expert users. 

Projects in this scenario might include projects to increase the amount of 
information available on-line, or improve the speed at which information becomes 
available. A low commitment project toward website evolvement might consist of 
highlighting finding aids on the homepage via a “Help” button in the left hand buttons 
column that links to the search feamre, A-Z, other indexes, and organizational charts. 
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Medium commitment 



The medium commitment option would require specific efforts to improve usability 
for all levels of users across the website. Medium commitment projects might include 
issuing guidelines for all www.ed.gov pages regarding including: 

• Discovery information for all links; 

• Further grouping and categorizing listed links on pages; 

• Warning users when links will take them off site and including homepage links; 

• Links to finding aids on every page 

• Reduction of “expert” terminology. 



High conmiitment 

A high commitment approach would require significant Department of Education 
resources for evolution away from current model in which “repurposing” of print content 
is the primary source of web content. Projects included under this level of commitment 
would include all the projects listed under the No/Low Commitment model and the 
Medium Commitment model. In addition, High Commitment projects might include: 

• Designing a “New User” button off the main home page which give new users a 
different and more intuitive interface for the site; 

• Designing sub-interfaces or “guides” for major user groups including parents, 
students and teachers which provide links to the most common resources used by 
those groups; 

• Posting FAQ files for each major page on the site, designing additional help 
features that assist new users in making use of Department resources (e.g. a how 
to write effective grants guideline page); 

• Offering all long documents in html format with links to sections and subsections 
to aid browsing; 

• Offering all documents in multiple formats. 



CONCLUSION 

This section presented an overview of the user assessment component of the 
project. The user assessment found that in general, www.ed.gov provides a wide array of 
useful information related to educational research, funding, programs and policies. The 
current website provides good access for experienced users. 

Suggestions from the user assessment centered around three main issues: the 
degree to which die site currently highlights its finding aids, the degree to which the site 
supported inexperienced, or novice users, and the degree to which the department will 
commit resources to evolving the site. 




i;o 



128 



Increasing site usability is not easy. Some recommendations are counter-intuitive 
or in conflict with each other, thus requiring an ongoing investment in usability testing 
geared to assessing improvement. It will also require the Department to make specific 
and directed resource commitments. Furthermore, it will require cooperation and 
coordination between all the disparate organizational entities that influence the content 
and presentation of www.ed.gov. Given these difficulties, it is tempting to push aside 
usability concerns. One should not forget however, that the end goal of the website, and 
of the Department as a whole, is to serve the public. Improving website usability will 
increase user satisfaction and allow the Department to fulfill its public service 
obligations. 
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VI. CONCLUSIONS AND GENERAL RECOMMENDATIONS 



The Department of Education’s website, www.ed.gov. is a rich information and 
service resource that is widely used both within the Department and by a variety of users 
external to the Department. It provides extensive information about the Department and 
its programs and services, enables users to interact efficiently with some programs (e.g., 
student financial aid), and serves diverse audiences. It has a rich web policy environment 
to facilitate Departmental decision-making in support of the development of the website. 
In relationship to other Federal websites, it is extremely well developed with a set of 
policies that go far beyond what other agencies typically have developed. But as the 
website has grown and developed, a number of issues - especially related to management 
and resource support — require attention. 

In recent years. Congress and the Administration have instigated numerous 
programs and initiatives in electronic information dissemination and in programmatic 
areas for which the department has responsibilities for implementation and oversight. In 
this context of significant growth in departmental responsibilities, there have been major 
demands to both respond to these responsibilities and to provide information and program 
services via the department’s website. The department’s efforts to move to a web-based 
environment is laudable, but it simply cannot maintain this effort without a reassessment 
of resource support and management structure - especially given the increased program 
initiatives required by congress and the administration. 



KEY FINDINGS 

As stated earlier, the Department has quite successfully “grown” its website creating 
a rich information environment within a rich policy and management context. However, 
the Department and the website are following a not untypical trajectory for organizational 
websites, a trajectory that, if continued, will have a number of negative consequences for 
the website. This trajectory is one in which resources and management structures do not 
keep pace with the demands made on the website and the staff associated with it. Most 
critic^ly, the ED website: 

• Lacks adequate staffing and resource support for current operations and this condition 
is likely to worsen dramatically as demands for Internet support grow. 

• Is outgrowing the capacity of the current website management structure. 

These two issues represent “make-or-break” issues for the website. Without additional 
resources and a reconsideration of the current management structure, the problems 
currently experienced will continue and likely grow worse. 
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In addition to these two critical problems, the website: 

• Needs to have expanded ongoing evaluation efforts associated with its use and 
growth. 

• May be attracting users who are not well served by the website structure, 
organization, or associated finding tools and services. 

• Is demonstrating some technical problems (such as non-standard coding, insufficient 
usage of META tags) which hinder user access. 

• Can better incorporate and comply with selected broader Federal policies and 
regulations. 

• Needs to move content beyond the current situation of reusing print content to a 
customer-centric site, which may present different information in ways that take 
advantage of the web’s capabilities. 

These conclusions result from the study team’s four-pronged analysis approach, detailed 

in the previous chapters. 



MANAGEMENT AND EVALUATION EFFORTS 

The most pressing concern for the Department regarding its website is the 
imminent outstripping of the capacity of the management structure and resources to meet 
the demands of the website. The team’s evaluation provides a picture of a structure and 
resource base severely strained by the current demands of the website. As the Department 
can expect these demands to continue growing, it is obvious that problems identified in 
this report will be exacerbated and likely other problems will emerge. 

The website and its management efforts are severely under-resourced. The study 
team believes that ED is facing a critical lack of staff in the area of website creation, 
maintenance, coordination across the various offices, and planning. There are 
insufficient staff in OERI and also in the Principal Operating Components associated 
with website activities to attend to website development and management, and these staff 
need additional training. The current centralized structure has worked effectively for 
oversight purposes to date, but again, this structure is now creating bottlenecks in website 
content processing, evaluation, and ongoing planning. Strategic planning activities have 
fallen by the wayside as staff concentrate on the abundance of day-to-day tasks that must 
be accomplished to keep the website up and running. 

Evaluation efforts have also necessarily remained minimal due to lack of resources. 
The contractor, while successful in maintaining content on the web, appears not to be 
fulfilling contractual obligations regarding submission of appropriate reports and data, 
and has not been evaluated using the performance measures in the contract. Feedback 
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from customers (via the website and other customer service activities), which was 
extensively used in early planning and design (and the first redesign) stages has not been 
integrated in a routine way into ongoing site planning activities. The evaluation 
techniques used in this report can provide the department with strategies that should be 
incorporated into ongoing website evaluation efforts. 

As web-based information services and resources grow throughout the Federal 
government, it is likely that more policy attention will be given to the role of Federal 
websites in meeting government-wide information and services dissemination guidelines. 
Although the Department of Education has clearly been working to integrate Feder^ 
policy into its web policies, it will need to continue to monitor policy developments in 
this area. Compliance with GPRA, EFOIA, records management guidelines, Y2K, and 
other policy initiatives will likely take on greater importance in the future and will require 
additional staff support. The Department will need to determine how best to assign 
responsibilities to insure agency compliance with these guidelines and requirements. 

Given these concerns, the Department should: 

• Clarify the primary purposes of the website and set priorities for what are the 
most important activities that should be supported via the website. 

• Reassess how website activities are resourced and immediately begin processes to 
increase that allocation. 

• Restmcture and examine the responsibilities of key individuals related to the 
website and develop strategies for eliminating potential bottlenecks and pressure 
points (specific recommendations are presented in the management section). 

• Implement a program of ongoing training for staff in basic website skills. 
(Training is required in the application contractor’s contract.) 

• Continue and expand log analyses. More sophisticated analysis software can be 
purchased. Procedures for addressing problems identified in log analysis should 
be implemented. Strategies to encourage webmasters to review and act upon log 
analysis reports should be developed. 

• Develop mechanisms for routinely integrating feedback from customers. 

• Establish responsibilities and procedures to monitor and comply with Federal 
policies that affect website development and use. 

• Develop a policy of ongoing website evaluation. 

The Department will continue to develop its web-based presence. It needs now to 
quickly address problems identified by this report and then to continue to assess how it is 
organized to support ongoing growth and development. In this process, the department 
should consider instimting more formal control and oversight over website development. 
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MOVING TOWARDS A USER-CENTERED SITE 

The team’s analysis of logs and user assessment point to a site which has highly- 
used areas, under-used areas, some technical problems which present possible difficulties 
for users, a number of difficulties related to searching and identifying specific 
information, and several design aspects that cater to the expert, knowledgeable user but 
may be less satisfactory to a novice user. This is a not untypical situation for an 
organizational site of this complexity and age. Most organizations first begin a website 
by recreating the organization’s stmcture and by providing information in essentially the 
same format as text-based versions of that information. 

To move the Department’s website to the next level of evolution requires that the web 
be perceived as a unique type of information tool that does not need to mimic 
organizational stmcture or other information dissemination strategies. By virtue of its 
ubiquity and its enabling of enhanced access for a diverse set of users, a website needs to 
shift in focus away from an organization’s perception of itself to the customers’ 
perception of it. A user-centered approach to design is necessary. 

A user-centered approach would mean that users are more proactively approached to 
determine their needs (both technical and in terms of information), their perceptions of 
website organization, terminology, ease of navigation, etc., and that site redesign would 
explicitly address those perceptions and needs. Additionally, knowledge about how users 
navigate websites, read online, etc. would be employed. For example, it may be useful to 
have different approaches to the website for those who are beginners versus those who 
are more sophisticated in their web skills and/or knowledge of the Department. It may 
also be important to consider “real time’’ user support and assistance. These general 
strategies change a site from one which “mimics’’ print publications to one that is a user- 
centered and user-responsive site (see Section EQ for details). 

The Department has already taken a number of steps in this direction. The team’s 
specific recommendations throughout the report and the general ones here should enable 
the Department to continue in this direction. To move towards a user-centered website, 
the Department could; 

• Explicitly address and specify the purpose(s) and audiences for the website. 

• Continue to tabulate the results of the online survey and incorporate mechanisms 
whereby the results are used by POCs and others to improve the website. 

• Establish a team (or several teams) of interested users to provide regular feedback to 
the website management on the site. 

• Undertake additional user studies of site terminology (for example, terms on the A-Z 
index could be assessed from a user perspective; one source of data are user terms put 
in on the search engines). 
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• Integrate website customer service activities with existing customer service activities. 

• Continue log analyses practices and extend using software that produces data such as 
that in Section IV of this report. 

• Develop policies/procedures for website content additions that incorporate user- 
centered research (e.g., on how people read online; typical user technical 
configurations) and which lead to a consistent “look and feel” as well as consistent 
content and its organization on the website. 

• Develop mechanisms beyond the IWG for the organization that would enable it to 
move beyond office-level website initiatives. 

The study team recognizes that shifting a website to a user-centered approach requires 
efforts and resources that go beyond the website itself. This process will not be simple or 
immediate but will require a philosophical shift in the organization that recognizes the 
increasing dominance of electronic and web-based information services and resources as 
the medium of choice. 



STRATEGIES FOR ONGOING EVALUATION 

A purpose of this study was to suggest ways in which the Department can 
continue a process for regular ongoing assessment of its website. The study team 
recommends that future assessment efforts continue to incorporate a four-pronged 
approach that incorporates log and transaction analysis, user assessments, management 
assessment, and policy assessment. As shown in this report, the combination of these 
approaches offers a powerful means to provide a comprehensive view for where a 
Federal website can be improved. 

In addition, the study team has provided the Department with detailed assessment 
techniques and data collection instmments. The appendices provide detailed descriptions 
of these methods and data collection instmments. Depending on the needs of the 
Department, the study team believes that many of these techniques and data collection 
instmments can be replicated or modified for future evaluation efforts. At issue is a 
commitment of time and resources to engage in such evaluation efforts. Nonetheless, the 
techniques offered in this report provide a basis for such efforts to be continued and 
refined. 



INCREASING ACCESS TO WEB-BASED RESOURCES AND SERVICES 

Clearly; the Federal government will continue to rely on the dissemination of 
government information and the provision of a range of services via a web and networked 
environment. Indeed, webmasters may not be able, at this time, to even foresee what 
those services might entail. The Department of Education has made significant strides in 
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realizing the vision of Vice President A1 Gore as quoted in the introduction to this report. 
That vision, however, requires that Federal agencies rethink how they “do business” and 
“provide services” in this evolving web environment. Part of this “rethinking” is 
allocating adequate resources and obtaining staff with the necessary skills to realize the 
vision. 



An issue that all Federal agencies must address is the degree to which they can 
simultaneously continue traditional information services and resources and move to and 
expand their web and networked delivery of information and services. In fact, most 
agencies are now at the point where it is exceedingly difficult to do more with less. The 
current situation at the Department of Education suggests that for at least the web-based 
services and resources, the Department will need to commit additional resources to do 
more. Thus, setting priorities for what traditional services might be dropped or 
redesigned for a web environment will be a key management task in the future. 

Improving the nation’s access to and use of Department of Education information, 
resources, and services, can be done via the web environment - indeed, the findings 
presented in this report show that such is clearly the case. The Department, however, 
should expect continued significant growth in the nation’s use of its web as more people 
become familiar with web services, more have computers and high-speed access from the 
home, and as more people expect to be able to conduct business via the web. With such 
rising levels of user demand and expectations, the Department should build on its existing 
strengths in web-based services and move quickly to respond to the concerns identified in 
this report. 
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Appendix II- 1, Department of Education Management Focus, Interview Guide Questions 

1 . PROVIDE PROJECT DEMOGRAPHIC QUESTIONNAIRE FOR RESPONDENT TO 
COMPLETE. (INCLUDED IN APPENDIX III MATERIALS) 

2. INTRODUCTION TO PROJECT 

A team at the School of Infonnation Studies at Syracuse University was contracted to undertake an 
evaluation of the Department’s website(s) (primarily www.ed.gov , and the various office sites) in order to 
provide a set of reconunendations associated with site design and management. Our approach is to 
investigate the website from a number of perspectives: a user perspective, transaction logging, a policy 
perspective, and a management perspective. Keith Stubbs identified you as someone who could help us 
better understand the current management of the site, and would have insight into issues that have surfaced 
in that management. 

Before we begin, are there any questions I can answer for you about the project? 

3. THE INTERVIEW 

We have some general areas of interest related to website management that we have found useful in 
previous work. There are probably other aspects that strike you as pertinent, and we are interested in those 
as well. 

A. Website Purpose and Audience 

1 . What is the purpose of the website? 

2. What are the audience groups for the website? 

(May also want to get at genesis of website-why was it developed, what planning process what used. 
Format and organization decisions which were made) 

B. Processes for content creation and getting on to web. 

I .What is the process for creating content and getting it on the website? Has that changed over the time the 
website has been in existence? What kind of content/information is presented on the website 

(Looking for procedural stages: origin of the data, verification of reliability of data, presentation of data to 
HTML coders, verification of content once online, etc, what’s working in the process, what isn’t, policy 
aspects such as what type of content is allowed) 

2. Describe the management structure surroimding the process described above. 



(This may be elicited by the above question. Looking for who does & 
decides what, was the process the result of planning or did it just 
evolve, etc.) 

3. How does the process of putting content online tie in with other processes 
of providing content - in print, etc? 

(e.g.: Do the online & print content go through the same process, or are 
Aey managed separately?) 

4. What mechanisms do you have to update content, technical aspects of the site and its content? 
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C. Feedback and User Questions and Comments 

1 . What mechanisms are in place to receive feedback (comments or questions) from users or potential 
users of the websites? 

2. Describe the process of dealing with feedback from the website. 



(Looking for procedural stages: who receives the feedback, how do they 
evaluate it, how do good ideas feed into the content provision processes?) 



D. Changes Due to Website 

1. How have the processes surrounding providing content in print (or other 
media) changed since the creation of the website? 

2. How have the processes surrounding the receipt of feedback from other 
media changed since the creation of the website? 

3. Have there been changes in job responsibilities/organizational structure that are due the website? 



E. Evaluation Processes 

Do you or does the organization have a model for determining the cost of maintaining the website? 

What processes are in place to evaluate the “success” of the website? 

What processes are in place to evaluated the “success” of the management of the website? 

F. Policy Aspects 

What policies in the organization or the Federal Government have helped or hindered website development 
and maintanence? 

What interagency website activities is the agency engaged in? 



G. Lessons Learned 

What do you feel are critical factors leading to the success of your website? 
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What do you feel is the best balance of technical/organizational/interpersonal skills or expertise needed to 
develop and operate a comples website? 

What is next for the website? 



H. Other Issues 



Are there other aspects of website planning, management, and evaluation that we should be aware of? 
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Appendix III-l, Analysis Of Department Of Education Policy 

Documents 
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Appendix III-l, Analysis Of Department Of Education Policy Documents 

This Appendix is organized by individual policy: each policy will be discussed in 
terms of each of the analysis criteria which present issues within the policy. However, in 
the case of the U.S. Department of Education World Wide Web Server Policy and 
Procedures, the analysis was conducted on each of the major sections of the dociunent 
due to its size and complexity. The sections are identified in the docmnent and in this 
analysis by roman numeral and section heading. This document, U.S. Department of 
Education World Wide Web Server Policy and Procedures, will be discussed first since 
the other policies spring from its directives. The version used for this analysis was the 
Microsoft Word version available for download from the URL below; page munbers 
referenced in this section come from the printed document. 

U.S. Department of Education World Wide Web Server Policy and Procedures 
http ://www . ed. go v/intemal/wwwstds.html 

I. Background 

Since this portion of the docmnent introduces the body of the docmnent, only 
minor points merit discussion here. First, the statement “The ED Home Page is the entry 
point to the official docmnent collection hosted on the ED-WWW Server, and originates 
at URL: http://www.ed.gov/” may require further clarification. If the docmnents found 
through the use of the above URL are the “official document collection,” does this 
actually mean that all docmnents found on the ED Website are considered “official”? If 
so, does this have implications for Records Management and for GRS 20? 

This section includes a paragraph stating what the U.S. Department of Education 
World Wide Web Server Policy and Procedures is not designed to be (p. 1, paragraph 6): 

This docmnent is not a training docmnent. It is not a complete style guide for 
composing HTML (hypertext markup language), nor does it provide a complete 
checklist of procedures for project leaders to follow to publish information on ED 
WWW servers, although certain procedural issues are addressed, (emphasis 
added) 

This initiates the question as to whether this document can (or should) point the reader to 
where such a checklist does exist. Along this line of consideration, though, this statement 
possibly contrasts with the fact that later in the docmnent there actually seems to be 
directions for project leaders to follow in order to publish information on the ED WWW 
servers. Both the sections titled File Transfer and Posting (p. 32) and Document 
Submission Policy and Procedures (p. 37) provide activities to be undertaken in 
publishing documents on the ED WWW servers. This statement may need modification. 

n. Definitions 
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Since this section gives key term definitions, it has not been analyzed according to 
the criteria outlined above. 

in. Administrative Roles and Procedures 

Several ambiguous phrases or terms are contained in this section. First, that the 
ED Webmaster will enforce the Department's standards by “assigning liaisons on an as- 
needed basis to assist POCs in posting documents to the ED-WWW Server” (p. 3, 
paragraph 2) presents the question of what qualifies as an “as-needed basis.” Is there a 
set of guidelines which detail what criteria or needs will necessitate the assigning of a 
liaison? 

A duty of each Assistant Secretary is to “ensure that web pages originating from 
his or her POC receive the proper reviews and approvals” (p. 3, paragraph 5). What are 
“proper reviews and approvals”? It might be helpful at this point to reference the areas 
later in the document which will explain the meaning of this statement. 

The use of the term “relevant” in the statement “A POC Home Page Editor who 
will be responsible for compliance with all relevant Departmental WWW standards” (p. 
3, paragraph 6) is open to interpretation. Which are the relevant standards and who will 
decide? 



It is not clear what is meant by the POC Home Page Editor responsibility of 
“organizing. . . main navigational pages” (p. 3, paragraph 6). Does this mean creating 
links, organizing categories, creating search engines, or perhaps other responsibilities? 

One of the duties listed for the Web Publisher is “responding promptly to 
customer inquiries” (p. 4, paragraph 1). What is considered prompt? Who will decide 
what is prompt? Who will evaluate and enforce prompt responses? 

Gaps in this section are found in the statement: “The ED Webmaster role is 
shared by the Office of Educational Research and Improvement (OERI) and the Office of 
the Chief Information Officer (OCIO)” (p. 3, paragraph 1). What are the specific 
position titles (if not names of people) within OERI and OCIO who share the webmaster 
role? Is there contact information occurring later in the document which can be 
referenced at this point? 

Further descriptive information would be helpful in imderstanding what ‘Topics 
A-Z and other applications of metadata” (p. 3, paragraph 2) means. Also, further 
information or a reference to find further information would be helpful in relation to the 
POC Home Page Editor's duty of “archiving and removing web pages” (p. 3, paragraph 
6). What are the guidelines (are there any?) for archiving and removing pages and who 
decides these guidelines? 

This section appears to indicate that the ED Webmaster will enforce the 
Department's standards by “assigning liaisons on an as-needed basis to assist POCs in 
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posting documents to the ED-WWW Server” (p. 3, paragraph 2). This seems vague. 
What conditions will trigger the assigning of an liaison? Similarly, the statement “Each 
Assistant Secretary is to ensure that web pages... receive proper reviews and approvals 
and meet required technical standards” (p. 3, paragraph 6) is to be done by designating a 
POC Home Page Editor and a Web Publisher. Will only this designation actually allow 
the Assistant Secretary to ensure “that web pages... receive proper reviews and approvals 
and meet required technical standards”? Shouldn't a review or evaluation using specific 
criteria occur? 

It is difficult to tell how the tasks undertaken by the Assistant Secretary, POC 
Home Page Editor, Web Publisher, and, for that matter, the ED Webmaster will be 
reviewed and evaluated to ensure compliance with the policies outlined in this section. In 
fact, a general sense of vagueness pervades this section with regard to the different job 
titles included in this section, who exactly does what, and the overall relationship of each 
job title to the others. In order for activities to be accomplished in accordance with the 
policies in this document it seems that carefully laid out, very clear explanations of who 
does what and the relationship of each actor to the others should be given. 

A process to collect user feedback, undertake modification of the policy, or 
eliminate the policy was not identified. 

rV. A. ED-Controlled Content 

The use of the term “organization” in the statement “The content of all pages on 
ED WWW servers shall be related to the function and mission of the organization” (p. 5, 
paragraph 1) is not clear. Does this mean the Department of Education or an office 
within the Department? Also, it would be helpful to reference a place (on the web or in a 
document) where the function and mission statements could be accessed. In the final 
sentence of the paragraph, the use of the term “link” raises questions, in light of the 
sections which follow, of whether what is meant in this instance is specifically an 
external link. 

It is indicated that “Every document (or document collection) must have an E- 
mail address” (p. 5, paragraph 6). Since documents and document collections were 
previously defined as separate entities (p. 2), is this meant to be an either/or statement? 

“POCs must ensure that all out-of-date information on the Web site is promptly 
removed, updated, or archived” (p. 6, paragraph 3) may raise the question as to what is 
considered “out-of-date.” While some material would obviously be of no use to users 
past a certain date, other out-of-date information may retain value. Should further 
direction be given to this point? 

This section does contain very helpful examples which assist clarity: for 

example, the example of a generic e-mail address (p. 6, paragraph continued fi-om 
previous page) and the examples in the list of quality assurance methods (p. 6, paragraph 
1 ). 
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Several areas may require more explanation. Remaining on the previous 
paragraph's topic, it is not clear what sorts of situations the term “policy implications” 
might include. Provision of an example or reference to guidelines elsewhere might be 
helpful. The reference to an “advance notice process” (p. 5, paragraph 5) seems to 
require further explanation. If it is a “process,” are there procedures or forms which must 
be followed or submitted? 

The statement regarding the fact that out-of-date information may need to be 
removed or archived and that the “party responsible for the dociunent or collection 
should determine whether to retain an electronic version corresponding to the outdated 
version” (p. 6, paragraph 3) again brings up the question of the possible need for 
guidelines. What should or can be removed or archived? Is archiving and removal of 
dociunents being left solely up to each “party responsible for the dociunent or 
collection”? Does this support an overall effort by the Department to centralize its 
operations? 

A possible inconsistency exists between the statement that the “central ED WWW 
server [is] operated by the OCIO” (p. 7, paragraph 3) and the statement that “Internet 
services will be developed and maintained centrally by OCIO and OERI” (p. 7, paragraph 
7). Does only OCIO maintain the ED WWW server? If so, should this be clearly spelled 
out in section “IH. Administrative Roles and Procedures”? If the server is being 
maintained by both OCIO and OERI, are there coordination issues between the two imits 
which need additional clarification? 

There are several enforcement methods indicated in this section. First, text under 
the Approvals subheader requires that new WWW documents gain approval from the 
sponsoring POC and from the OPA if policy implications exist. In addition, an approval 
process for purchasing additional servers is outlined (p. 8, paragraph continued from 
previous page) to prevent POCs from purchasing servers on their own. 

A process to collect user feedback, undertake modification of the policy, or 
eliminate the policy was not identified. 

rV. B. ED-Sponsored Content 

Several items which may be ambiguous appear in this section. The text under the 
subheader Statement of Purpose (p. 8, paragraph 1) is ambiguous in that it does not 
explicitly state that a statement of purpose is required. Is such a statement required? 
Next, it is not clear what “coordination of all ED-sponsored pages” (p. 8, paragraph 2) 
includes. Is this term all-encompassing as far as authoring, posting, and maintaining 
websites goes or is it more limited? While the policy indicates that the “POC and the 
responsible Home Page Editor will decide how closely the site needs to be monitored” (p. 
8, paragraph 2), who will actually be doing the monitoring? The Home Page Editor? 
Finally, the policy states that one factor which will contribute to the level of POC 
monitoring is the “contractor/grantee track record for responsibly managing the site” (p. 
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8, paragraph 2). How long is the time period for the track record and what specific 
factors make up the track record? 

Further information may be desirable in this section in relation to the provision of 
an explicit statement regarding the award which funds the project (p. 8, paragraph 1). 

Lastly, it seems that the POC and Home Page Editor are responsible for enforcing 
the policies in this section. Are they? If so, what are the explicit procedures for such 
enforcement? A process to collect user feedback, imdertake modification of the policy, 
or eliminate the policy was not identified. 

rV. C. External Content 

This policy contains the statement that the “ED Webmaster reserves the right to 
approve or disapprove a request to link to non-ED sites” (p. 9, paragraph 1). However, it 
is imclear whether POCs actually have to make a formal request to link to non-ED sites. 

If they do, what is the process for such a request? What specific criteria will the ED 
Webmaster, in consultation with OGC, be using to approve or disapprove link requests? 
Further, what is the purpose of developing a POC policy for linking to external sites if 
links must be approved by the ED Webmaster? In addition, the policy directs that 
prominent disclaimers be displayed in “various locations of the POC web site” (p. 9, 
paragraph 1). The meaning of “various locations” is ambiguous. 

This policy will apparently be enforced by the ED Webmaster in consultation 
with the OGC (p. 9, paragraph 1). Without detailing the specific criteria which will be 
used to approve and disapprove links, though, actual enforcement provisions are vague. 

Again, a process to collect user feedback, undertake modification of the policy, or 
eliminate the policy was not identified. 

V. Navigation/Organization 

This policy appeared ambiguous fi’om an overall standpoint. It was not explicitly 
clear firom the text what the policy actually was. In other words, what exactly is to be 
done in response to this policy and by whom? Adding to the ambiguity was the use of 
the term “coordinated” in the statement “ED-sponsored WWW materials must be 
coordinated with the main ED WWW server” (p. 10, paragraph 3). What activities are 
included by “coordinated”? 

Further information would be helpful in regard to the process for reciprocal 
linking (p. 10, paragraph 4). How is this accomplished? Who does the linking, who 
needs to receive the linking information, and is there a notification process for new 
material which needs to be linked? 

In addition, the third paragraph references ED-sponsored WWW materials and 
does not mention ED-controlled materials. As defined in a previous section, IV. 
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Content, ” ED-controlled information is that information on WWW servers “operated or 
directly representing Department offices,” while ED-sponsored information is 
information on WWW servers “operated by or directly representing organizations or 
individuals under Department funding” (p. 5, paragraph 1). Is the lack of reference to 
ED-controlled materials in this section material? How will ED-controlled materials be 
link ed with the appropriate pages on the ED WWW server? Should this page reference 
the Cross-links Table on pages 22 through 24? 

There is a slight inconsistency evident in the text of this policy in that while 
navigation is specifically listed as a Home Page Editor duty previously in this document 
(p. 3, paragraph 6), no mention of the Home Page Editor is made in this section. 

Neither statements regarding how the policy is to be enforced nor how to collect 
user feedback, imdertake modification of the policy, or eliminate the policy were 
identified. 

VI. Compliance Issues 

This section contains several examples that contribute to the clarity of the policy. 
Examples include those web applications which may increase security risks (p. 11, 
paragraph 6), a sample copyright notice (p. 12, paragraph 4), and examples of identifying 
information (p. 13, paragraph 1). 

Amb iguous phrases occur in a number of places in this policy. First, the POC 
Home Page Editor and Web Publisher are directed to coordinate with OERI “to ensure 
proper linkage when. . .substantial changes are made to an existing collection” (p. 11, 
paragraph 2). What would qualify as substantial changes? In the next paragraph, the 
policy indicates that POC Home Page Editors and Project Monitors are responsible for 
the “quality and functionality of their web pages” (p. 11, paragraph 3). What exactly do 
the terms “quality and “functionality” mean? Definitions of these terms may be helpful. 

Similar questions may be asked of the use of “fi’om time to time” in the paragraph 
indicating that Web Publishers should evaluate the needs of their projects (p. 13, 
paragraph 2). How often, specifically, should Web Publishers be evaluating such needs? 
And, what is meant by the use of “appropriate adjustments” in reference to what sort of 
changes Web Publishers should be making (p. 13, paragraph 2)? Would it be helpful to 
define what sorts of adjustments Web Publishers should consider? 

The paragraph imder the subheader Coordination (p. 11, paragraph 2) seems to 
duplicate the intent of the previous section, “V. Navigation.” Yet, this paragraph 
provides further information to assist in coordination in the form of contact details. 
Should this have been included in the previous section, “V. Navigation,” instead? 

Several subsections of this policy, “VI. Compliance Issues,” may benefit by some 
further explanation. For example, the text imder the subheader Security/Privacy may be 
helped by a description of what sort of consultation process is required before using web 
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applications considered security risks (p. 11, paragraph 6). Do the ED Webmaster, 
Departmental Security and Privacy Act officials, and the POC Home Page Editor all need 
to be consulted before using one of the mentioned web applications? In what order? 
What information needs to be presented? 

Regarding the subsection Government Information Locator Service (GILS) 
Compliance (p. 12, paragraph 6), it may be useful to provide further brief information in 
this policy as to when GILS records should or should not be made, what the process is for 
developing the records, and how the records are submitted and to whom. Perhaps this 
subsection should reference the appropriate GILS Policy and Guidelines at NARA and on 
other websites. 

The subsection Usage Monitoring (p. 13, paragraph 1) may be enhanced by 
providing information as to how Web Publishers should review and analyze usage 
reports, how often usage reports should be analyzed, and with what criteria the reports 
should be analyzed. Finally, website changes which would qualify as “fiivolous and 
arbitrary” (p. 13, paragraph 4) could be clarified through the provision of examples. 

This section may add to the inconsistency of previous information given 
regarding which Department of Education office is doing what activities. This section, 
“VI. Compliance Issues,” indicates that OERI maintains the ED WWW server (p. 11, 
paragraph 2). However, it was earlier indicated that: “The ED Webmaster has the 
primary responsibility for administration of the ED-WWW Internet Server. The ED 
Webmaster role is shared by the Office of Educational Research and Improvement 
(OERI) and the Office of the Chief Information Officer (OCIO)” (p. 3, paragraph 1); that 
the “central ED WWW server [is] operated by the OCIO” (p. 7, paragraph 3X aiid that 
“Internet services will be developed and maintained centrally by OCIO and OERI” (p. 7, 
paragraph 7). Is it reasonable to have these various activities in different offices — are 
these activities really being performed by different offices — and will other staff 
understand the different roles? 

Another inconsistency arises fi'om the text under the subheader Security/Privacy 
which indicates that the reader should “see References for requirements and sample 
disclaimers” (p. 12, paragraph 1) for privacy or security issues. Yet, in the section “VTI. 
References” (p. 14), there appears to be no such information. 

Enforcement of this policy, at least with regard to certain web applications which 
implicate privacy considerations, will be accomplished by the ED Webmaster, 
Departmental Security and Privacy Act officials, and the POC Home Page Editor. 
Further enforcement efforts are not explicit. 

User feedback and processes for policy modification and policy elimination are 
not explicitly addressed. 

Vn. References 
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The text in this section indicates that the section is “not an exhaustive list” of 
statutes and issuances which “must be complied with in the collection, dissemination, 
access, preservation, and privacy of Government information resources” (p. 14, 
paragraph 1). However, is there an exhaustive list stating exactly what statutes and 
issuances must be complied with somewhere else? If so, it may be helpful to reference 
such a list in this section. 

The final paragraph states that OMB is “encouraging federal agencies to sue the 
Web in a manner consistent with the agency’s mission. . .” (p. 14, paragraph 3), among 
other provisions. Is there a statement, guideline, or policy somewhere fi-om OMB which 
would clearly spell out OMB’s requirements or expectations? Reference to such a 
statement here would be helpful information. 

Since this section contains references to statutes, enforcement of these statutes 
will be contained within their text. There is, however, again no explicit methods for user 
feedback, policy modification, or policy elimination. 

Vin. A. Technical Standards and Guidelines 

This section does aid the user’s understanding of its directives by the inclusion of 
several helpful examples which illustrate concepts. For instance; examples of 
proprietary file formats are given (p. 15, paragraph 3); examples of good URL names are 
given (p. 17, paragraph 2); and examples of logical and physical tags are given (p. 29, 
paragraph 4). 

Several areas may be helped by further clarification. First, this section states that 
portable document formats should not be used as the primary format unless “converting 
the material to HTML is not feasible” (p. 15, paragraph 5). What is the threshold for 
feasibility in this instance? Are there guidelines as to document length or complexity 
which would help the user to know when he or she should be converting to HTML? This 
same question arises later in the directions for proprietary formats: “Proprietary formats. 

. . should only be used if; (a) conversion to one of the above formats is not feasible” (p. 
16, paragraph continued fixim previous page). Again, what is the threshold of feasibility? 

Another paragraph indicates that “Data files shall be accompanied with adequate 
documentation of the file content and structure” (p. 16, paragraph 2). Without any 
explicit guidelines, the use of “adequate” is open to subjective interpretation. 

Under the subheader File Size a statement directs that “either a PDF or a 
compressed file in ZIP format should be made available” to users when a document 
consists of more than ten files (p. 18, paragraph 1). Should the PDF file or compressed 
file be made in addition to or instead of the more recommended HTML file (see p. 15, 
paragraph 4)? It is unclear from the text. 
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A paragraph under the subheader To Download Files makes a reference to 
sections titled “File Formats and Large or Complex Documents” (p. 21, paragraph 2). 
Does this mean, in fact, the subsection titled File Size? 

In the paragraph for the subheader Titles, the policy directs that “Every page will 
have a unique title” (p. 25, paragraph 2). How will anyone know what titles have already 
been used? Is there a list of all titles employed by ED web pages somewhere which 
could be referenced here? 

The policy under the subheader Accuracy/Code Checking states that document 
“spelling and syntax should be checked, and the links should be verified” (p. 32, 
paragraph 2). Whose job is it, specifically, to do these tasks? Further, should this be 
done or must it be done? Along the same lines, a later paragraph vmder the subheader 
Adding New Files directs that “two hard copies and a disk containing an electronic 
version” of the document be provided to OERI or the ED Web Contractor (p. 32, 
paragraph 5). Whose job is it to provide these items? 

An apparent contradiction exists between the directive that Zip, Jaz, and SyQuest 
disk formats are acceptable (p. 33, paragraph continued from previous page) and the later 
statement that SyQuest cartridges, Jaz and Zip drives, magnetic tape, etc. are to be 
avoided (p. 38, note 2). It is unclear which format is allowable from these statements. 
Further, the use of “etc.” in note 2 on page 38 introduces ambiguity and should have 
specific items substituted for it. 

The body and intent of the instractions for the subsection File Transfer and 
Posting (beginning at p. 32, paragraph 5) is very similar to the section “Vm. B. 
Document Submission Policy and Procedures” (p. 37). It is not clear why these 
seemingly duplicative policies are offered in separate sections. It may make sense to 
consolidate these two sections into one, thereby eliminating the possibility for 
inconsistencies. 

Further explanation would be helpful in a number of places. While the policy 
indicates that the user should “send suggested revisions/additions to the ED Webmaster” 
(p. 15, paragraph 1), contact information for the ED Webmaster would be helpful. 

References are made to “progressive JPEGs” (p. 15, paragraph 4), “self-extracting 
archive files” (p. 16, paragraph 4), and the “interlacing technique” (p. 28, paragraph 
continued from previous page). What do these terms mean? Will the average user 
already know? 

The text under the Cross-links Within ED Site subheader speaks to determining 
when links should be added, revised, or removed (p. 22, paragraph 1). Is there a process 
for adding, revising, or removing links? Once it is determined friat something should be 
done, who does it? 
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Reference is made again to keeping an old document online for “archival or 
reference purposes” (p. 34, paragraph 3). What are the guidelines for deciding whether to 
keep a document online or to archive it? It would be helpful to reference such guidelines 
here. 



An inconsistency arises in the directions given for the presentation of e-mail 
addresses for users to contact the Department. First, in section “IV. A. ED-Controlled 
Content,” it is stated that “Every document. . .must have an e-mail address or link to an e- 
mail form” (p. 5,paragraph 4). Yet in the section currently being analyzed, there is text 
stating that “Mailto links will be used to route e-mail to Department mailboxes” (p. 21, 
paragraph 1) and later that every document will include a standard footer having an “e- 
mail address or initials of responsible party with mailto link” (p. 25, paragraph 8). 
Should all web pages have mailto links or is it acceptable for them to only have e-mail 
addresses? 

This section, “Vni. A. Technical Standards and Guidelines,” also does not 
include any reference to the need to submit the form “Request to Post Files on ED WWW 
Site.” Later, in section “VIII. B. Document Submission Policy and Procedures,” the 
submission of this form is indicated as necessary. 

The subheader References (p. 35) contains references to documents and their 
URLs. The first document referred to. World Wide Web Federal Consortium Home Page 
Guidelines and Best Practices, gives a URL 

(http://skydive.ncsa.uiuc.edU/cybrary/2/guidelines) which still exists but contains out of 
date information. The “official” version of this document now resides elsewhere and 
may soon be updated further. There is no indication at the address above that the official 
document is now elsewhere. ED is then linking to an older version of the document that 
no longer matches the current version of the document. 

This will likely be a problem for federal agencies in general. There is currently 
no real guidance as to when federal agency web pages can or should be removed and 
there seems to be a tendency to leave items in place even when they are no longer 
current. This tendency works at cross-purposes wiA the need to maintain the most up-to- 
date information possible. One agency might link to another agency’s old information 
without ever realizing that this information is no longer current. As long as the URL 
continues to function, and there is no indication on the old page that it is no longer 
current, how will federal agency webmasters know for sure that all their links are up-to- 
date? 



There are no explicit statements indicating how any of the policies contained 
within this section will be enforced. For instance, the use of client-specific markup is 
prohibited (p. 29, paragraph 2), but who is going to check to make sure this is adhered to? 
Later it is directed that pages be tested using Lynx and validated using Bobby (p. 30, 
paragraph 8). Who needs to know that this has been done and how will they know? 
These are just a few examples which illustrate the lack of enforcement present in this 
policy. 
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There is an explicit solicitation for user feedback in this document: “Please send 
suggested revisions/additions to the ED Webmaster.’’ (p. 15, paragraph 1). However, it 
may be helpful to indicate in the paragraph on “Dead Links’’ (p. 24, paragraph 1) that 
feedback should be solicited from the external users of the website on this issue. 

Vm. B. Dociunent Submission Policy and Procedures 

This section is very similar to the previous subsection. File Transfer and Posting 
(p. 32, paragraph 5) with the exception that “VIII. B. Document Submission Policy and 
Procedures” is more detailed. Which is to be followed exactly? Duplicating procedures 
leaves room for ambiguities and inconsistencies to arise. For example, this section does 
not mention the need to submit two hard copies and a disk of the material to be posted to 
a website which was required in the subsection File Transfer and Posting (page 32, 
paragraph 5). This is problematic. 

Again, there are no specific enforcement guidelines in this section, neither are 
there guidelines as to how the policies in this section can be commented on, modified, or 
eliminated. 



ED-WWW Server and Site Problems Customer Support Process 
http://inet.ed.gov/intemal/customer.html 
http://inet.ed.gOv/offices/OCIO/intemet/inet5.html 

This policy is straightforward and is unambiguous. It contains a specific example 
of an e-mail acknowledgement to a complainant, aiding in its understandability. The 
version at <http://inet.ed.gov/offices/OCIO/intemet/inet5.html> also contains a mailto 
link for question, comments, and suggestions, facilitating user feedback. There are, 
however, no statements as to how directives contained in this policy will be enforced or 
by whom they will be enforced. 

Establishing Links to External Organizations 
http://www.ed.gov/intemal/xtmlink.html 

Several portions of this policy contain ambiguous wording. First, the statement 
“If the Department’s main page of links to external sites is not used, collections of links 
to external sites shall be accompanied by a statement of the scope of the collection and 
the criteria for its inclusion” (emphasis added, paragraph 5) is awkward wording. Does 
the italicized portion of this sentence actually mean “the criteria by which external links 
are evaluated for inclusion”? 

Paragraph 5 also contains the text, “Unless the stated scope and criteria are well- 
defiined and limited, the sponsor has no grounds for refusing such requests.” Are there 
guidelines as to what would be considered “well-defined and limited?” These terms 
seem to be open to interpretation. 
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In paragraph 7, the policy states that “The collections provided at 
http://www.ed.gov/EdRes/ should be provided as gateways for our customers to non-Ed 
sources of information.” Does this mean that all document collections should provide 
links to this other page? This paragraph continues and makes a reference to the 
“Department’s ‘Other Sites’ page.” Is “Other Sites” the title of the page that will be 
found at the URL <http://www.ed.gov/EdRes/>? This policy would be clearer if the 
meaning of “Other Sites” was made explicit. 

Finally, the policy contains the statement, “The IWG shall reserve the right to 
review proposed external link collections before they are implemented” (paragraph 8). Is 
there a process by which external link collections need to be submitted to the IWG for 
such a review? Is there a specific time period for this submission and review? 

This policy contains several statements which seemingly contradict policies in the 
U.S. Department of Education World Wide Web Server Policy and Procedures. First, the 
policy Establishing Links to External Organizations directs that “When external links 
appear on a page but are not its major focus, the standard Disclaimer icon shall appear 
next to each external link or group of links; the icon shall link to the disclaimer statement 
at http://www.ed.gov/disclaimers/exit.html” (emphasis added, paragraph 3). The main 
ED policy document, however, states “When external links appear on a page but are not 
its focus, the standard Exit icon (http://www.ed.gov/disclaimers/exit.gif) shall appear next 
to each external link or group of links. The icon shall link to the disclaimer statement at 
http://www.ed.gOv/disclaima:s/exit.html” (emphasis added, p. 20, paragraph 3). It seems 
that individual policies should be consistent with the main policy document, not the other 
way around, and that the icon referred to in Establishing Links to External Organizations 
should actually be the Exit icon. 

Another apparent contradiction arises in this policy’s statement “The IWG shall 
reserve the right to review proposed external link collections before they are 
implemented” (paragraph 8). Yet the main policy document, U.S. Department of 
Education World Wide Web Server Policy and Procedures, states “The ED Webmaster 
reserves the right to approve or disapprove a request to link to non-ED sites in 
consultation with OGC” (p. 9, paragraph 1). Who is actually reviewing external links — 
the IWG or the ED Webmaster, in consultation with OGC? 

Inconsistencies also occur in this document in relation to the main policy 
document. This policy directs that for pages whose major purpose is to link to external 
sites, the standard Disclaimer of Endorsement statement 
(http://www.ed.gov/disclaimers/endorse.html) shall be included on the page (paragraph 
3). However, the U.S. Department of Education World Wide Web Server Policy and 
Procedures indicates: "Pages whose focus is linking to external sites shall include or link 
to the text of the standard Disclaimer of Endorsement statement 
(http://www.ed.gov/disclaimers/endorse.html) (p. 20, paragraph 3). Is the Disclaimer of 
Endorsement statement to be included or can it be linked to? 
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Next, Establishing Links to External Organizations provides the option of using 
the Department's scope and criteria statement: a “scope and criteria statement for the 
Department's main collection of links to external sites (http://www.ed.gov/EdRes/) is 
available at http://www.ed.gov/EdRes/edrespol.html” (paragraph 4). If this is not used, 
then the “collections of links to external sites shall be accompanied by a statement of the 
scope of the collection and the criteria for its inclusion” (paragraph 5). This option is not 
given in the U.S. Department of Education World Wide Web Server Policy and 
Procedures. Here it is indicated only that “a statement of the scope of the collection of 
external links and the criteria for inclusion” shall be given (p. 20, paragraph 3). If there is 
to be an option of linking to a general ED scope and criteria statement, shouldn’t this 
option be contained in the main policy document? 

Finally, the U.S. Department of Education World Wide Web Server Policy and 
Procedures states that “The decision to include a link to an external source should be 
based on the statement describing the purpose of the document collection” (p. 9, 
paragraph 1). There is no such explicit statement in Establishing Links to External 
Organizations. It seems important to indicate explicitly what linking decisions should be 
based upon. 

Interestingly, Establishing Links to External Organizations seems to expand the 
basic directives of the main policy document with no evidence of explicit support for 
such expansion from the main document. For example. Establishing Links states, “As 
such, POC Web Publishers and Home Page Editors shall rely upon the Department’s 
centrally maintained collections of links wherever possible, rather dian compiling similar 
lists of links which may be duplicative, incomplete, and less accurate than the centrally- 
maintained Master lists” (paragnq)h 7). No such directive was found in the U.S. 
Department of Education World Wide Web Server Policy and Procedures. It does not 
seem like a sound rationale to allow policies deriving from a main policy document to go 
further than the original document. 

Other examples of this occur in Establishing Links. For instance, there is the 
statement “If the IWG decides that the proposed collection should not be supported by 
the Department’s core fimding, the sponsoring organization may choose to pay for the 
effort with its own fimds, either through the central contract or other means” (paragraph 
8). No such directive was found in the U.S. Department of Education World Wide Web 
Server Policy and Procedures. Also, Establishing Links states, “The collections provided 
at http://www.ed.gov/EdRes/ should be provided as gateways for our customers to non- 
Ed sources of information” (paragraph 7). No such directive was found in the U.S. 
Department of Education World Wide Web Server Policy and Procedures. 

Although it is understood from this policy. Establishing Links, that the IWG will 
review proposed link collections, this enforcement process is not detailed or specific. In 
addition, although there are hyperlinked initials in the web page’s footer, Aere is no 
explicit request for user feedback or explicit process in which this policy can be modified 
or eliminated. 
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Collection Scope and Criteria for Inclusion on Main ED Page of Links to Other Online 

Educational Resources 
http://www.ed.gov/EdRes/edrespol.htm 

This policy is for the most part clear and unambiguous. Language which could 
possibly be construed in multiple ways is the term “authoritative” which appears in 
“overall quality and authoritative value of the website” (paragraph 5) and later under the 
subheader General Catalogs and Subject Trees on Education in reference to “authoritative 
Web collections.” What exactly qualifies as “authoritative”? Is there a generally 
accepted definition of this term somewhere? 

The criteria for evaluating websites of external organizations appear subjective 
and rather vague (paragraph 5). For instance, what does “overall quality and authority” 
mean exactly? What qualifies as relevant and useful content? To what degree does a 
proposed link’s material have to complement existing information? And, who is judging 
this? The ED Webmaster, the responsible ED Office (POC Home Page Editor), or 
someone else? 

In addition, this policy makes the assertion that “ED will not link to sites that 
exhibit hate, bias, or discrimination” (paragraph 6). However, is it clear whether 
everyone has the same idea as to what constitutes hate, bias, or discrimination? Do 
further references exist which could clarify what materials qualify as exhibiting hate, 
bias, or discrimination? 

This policy seems to contradict earlier directives regarding who will approve or 
disapprove external links. Here, in Collection Scope and Criteria for Inclusion on Main 
ED Page of Links to Other Online Educational Resources, the policy says that “ED 
reserves the right to deny or remove links” (paragraph 6). In Establishing Links to 
External Organizations, the policy directs that “The IWG shall reserve the right to review 
proposed external link collections before they are implemented” (paragraph 8). In the 
main policy document, U.S. Department of Education World Wide Web Server Policy 
and Procedures, the policy states that “The ED Webmaster reserves the right to approve 
or disapprove a request to link to non-ED sites in consultation with OGC” (p. 9, 
paragraph 1). Evidence of these contradictions in two policies which should spring firom 
the main policy document indicate that care should be taken to ensure that all later 
policies are consistent with the main policy document. 

An apparent inconsistency within this policy. Collection Scope and Criteria for 
Inclusion on Main ED Page of Links to Other Online Educational Resources, arises 
between the indication in paragraph 3 that links can be made to “national and state 
education agencies and associations” and the statement under the subheader National 
Educational Associations and Organizations that “ED does not link directly to local and 
state associations or organizations.” It appears that the language in paragraph 3 should be 
clarified to read “national education agencies and associations and state education 
agencies.” 
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There are no explicit enforcement statements contained within this policy. Also, 
although there are hyperlinked initials in the web page’s footer, these initials do not 
explicitly request user feedback. There is no explicit manner in which this policy can be 
modified or eliminated. 



Disclaimer of Endorsement 
http://www.ed.gov/disclaimers/endorse.html 

Disclaimer of Endorsement contains text which is to be used in specific instances 
specified in other policy documents: for example. Establishing Links to External 

Organizations (http://www.ed.gov/intemal/xtmlink.html)and U.S. Department of 
Education World Wide Web Server Policy and Procedures (p. 20, paragraph 3) both 
reference this document. 

The text of the Disclaimer of Endorsement is clear, imambiguous, and does not 
seem to present any contradictions or inconsistencies. There is no enforcement necessary 
with this document. No modification or feedback options are present, although it may be 
helpful to include them. 



Copyright Status Notice 
http://www.ed.gov/disclaimers/copyright.html 

Several aspects of this notice raise questions. First, the text of this notice contains 
the statement, “Click on the ‘Back’ button on your browser to return to the page with an 
external link and then click on the text link to follow it.” However, it is not clear that all 
users arriving at this page would have come fi"om a page with the possibility to follow an 
external link. Couldn’t this notice appear on any ED pages? 

It is also unclear as to when and where this notice should be used. In the main 
policy document, U.S. Department of Education World Wide Web Server Policy and 
Procedures, the concept that ED material is not copyrighted is discussed (p. 12, 
paragraph 3), but no mention is made of using a notice like the one here. The main 
policy document needs to contain a specific statement referencing this dociunent and 
explicit indications as to when it should be used. 

No modification or feedback options are present, beyond hyperlinked initials in 
the web page’s footer, although it may be helpful to users to include such options. 

Privacy and Security Notice 
http://www.ed.gov/disclaimers/privacy.html 

At the bottom of the page of this notice italicized text contains the statement, 
“Click on the ‘Back’ button on your browser to return to the page with an external link 
and then click on the text link to follow it.” However, it is not clear that all users arriving 
at this page would have come firom a page with the possibility to follow an external link. 
For instance, it is possible to access this page via the link “Disclaimers and Notices” on 
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the ED home page (http://www.ed.gov). Therefore, if the user clicked the back button 
having come from the ED home page, the reference to an external link would not make 
sense. 



This notice also contains the text “Information presented on the WWW site is 
considered public information and may be distributed and copied” (paragraph 2). 
However, this duplicates material in the Copyright Status Notice 
(http://www.ed.gov/disclaimers/copyright.html). In addition, this statement does not 
seem to fit with the context of this notice, which is privacy and security not copyright 
issues. 



It may be helpful to users to provide more information through this document in 
the form of hyperlinks. For example, could a link be provided for the user to the 
pertinent portion of the National Archives and Records Administration General Schedule 
20? Could links be provided to the Computer Fraud and Abuse Act of 1986 or other 
pertinent websites related to security and privacy policy? 

Also, the notice states that “The Department posts a Privacy Act notice at those 
places on this Web site. . . where ED needs to collect any individually identifiable 
inf ormation” (paragraph 5). It may be helpful to provide the text or icon of the actual 
notice in this document so the user will know exactly what to expect or look for. 

In addition, as in the case of the Copyright Status Notice, there is no clear 
reference in the main policy document, U.S. Department of Education World Wide Web 
Server Policy and Procedures, as to when and where the Privacy and Security Notice 
should be used. In the main policy document under the subheader Security/Privacy (p. 
12, paragraph 1) the policy states that certain web applications will need to have: 
“appropriate warnings, disclosures and/or disclaimers [] openly displayed.” A reference 
should be made at this point in the main policy document to this specific document. 
Privacy and Security Notice, and guidelines should be given in the main document for the 
notice’s use. 

There are no enforcement statements attached to this notice since it is a notice and 
not a policy. There is an explicit statement which solicits user feedback and includes a 
mailto link. 



http://inet.ed.gOv/offices/OCIO/intemet/inetl.html 

The text contained in this document is informational and provides links for further 
information. This document is therefore not policy in the sense understood here and does 
not lend itself to analysis. 

Guidelines for Reviewing ED Web Sites 
http://inet.ed.gOv/offices/OCIO/intemet/inet7.html 
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There is no explicit reference in the main policy document, U.S. Department of 
Education World Wide Web Server Policy and Procedures, which provides for this type 
of review. Guidelines for Reviewing ED Web Sites, however, contains directions as to 
how the policies in the main document will be enforced. A reference in the main 
document to this policy. Guidelines for Reviewing ED Web Sites, should be made. 

Guidelines for Reviewing ED Web Sites does contain a statement soliciting user 
feedback along wi& a mailto link. 



Review Checklist 

http://inet.ed.gOv/offices/OCIO/intemet/inet8.html 

The Review Checklist contains minor examples of ambiguous language. First, 
paragraph 1 states that “ED sites should be reviewed frequently” and paragraph 3 
indicates that information on main web pages should be reviewed “regularly.” How often 
is frequently and regularly? It would reinforce the concept to state explicitly here how 
often web pages should be reviewed, especially since Guidelines for Reviewing ED Web 
Sites indicates that websites should be reviewed at least semi-annually. Also, the policy 
states that “open-standard” HTML is to be used for coding web pages (paragraph 7). 
What does this term mean? 

Page references in the Review Checklist back to the main policy document, U.S. 
Department of Education World Wide Web Server Policy and Procedures, do not match 
the actual pages of the main document. For example: 

• Links to E-mail addresses is discussed in the main document on pages 5 and 21 

• Timeliness/Currency of ED-controlled content: page 6 

• Principal Office Home Pages, Program and Project Home Pages: page 7 

• Links to Home Page: page 20 

• Navigation/Organization: page 10 

• Cross-links within ED Site: page 22-24 

• Accessibility: page 1 1 

• Considerations for Users with Disabilities, Character-only, or Low-Bandwidth 

Access: page 30 

• Section VIE. Technical Standards and Guidelines: page 15-32 

• Client Feature Variations, Client-Specific Markup: pages 28-29 

• External Content: Approval, Context: pages 8-9 

• Approvals for ED-Controlled Content: page 5 

• Approvals for ED-Sponsored Content: page 8 

• Copyright and Fair Use: page 12 

Another inconsistency arises from the statement directing the reviewer to v-aify 
that “There is an E-mail liiik (‘mailto’) on the main pages of a document collection” 
(emphasis added, paragraph 2). The U.S. Department of Education World Wide Web 
Server Policy and Procedures contains the statement, “Every document (or document 
collection) must have an E-mail address or link to an E-mail form” (emphasis added, p. 
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5, paragraph 6). The policy statements within this document, Review Checklist, must be 
consistent with the main policy document. 

Another example of inconsistency between this dociunent and the main policy 
dociunent exists. The statement in the Review Checklist directing reviewers to check that 
“All pointers to external sites are. . . consistent with the communications goals of the 
Department” (paragraph 9) is not consistent with the statement in the U.S. Department of 
Education World Wide Web Server Policy and Procedures that “The decision to include a 
link to an external source should be based on the statement describing the purpose of the 
document collection (see IV. A.)” (p. 9, paragraph 1). The language for a policy which 
appears in multiple documents must be consistent and should rely on the language 
contained in the main policy document. 

Since the purpose of this dociunent is enforcement, this criterion is well discussed 
here. Also, questions, comments, and suggestions are solicited and a mailto link is 
provided for this purpose. 

Procedure: Request for New External Web Server/Site 
http://inet.ed.gOv/offices/OCIO/intemet/inet3.html 

One apparently ambiguous statement in this policy ensues from the statement that 
proposals be submitted to allow IWG members “sufficient turnaround time” for review of 
the proposals (paragraph 5). Similarly, “adequate lead time is necessary so that principal 
offices can work with the Office of Educational Research and Improvement” (paragraph 
5). Exactly how much time is indicated by these two phrases? 

In addition, there is no explicit reference in the main policy document, U.S. 
Department of Education World Wide Web Server Policy and Procedures, to this 
document. A reference should appear in the main document since Procedure: Request 
for New External Web Server/Site fleshes out the policy contained in the main policy 
document. 

This policy will apparently be enforced by IWG representatives and the IWG. 
Questions, comments, and suggestions are solicited for this policy and a mailto link is 
provided for this purpose. 

Form: Request for New External Web Server/Site 
http://inet.ed.gOv/offices/OCIO/intemet/inet4.html 

This is a form which is to be submitted when requesting new external web servers 
or sites. The one ambiguity which may exist within this document is the use of “SOW 
in the last sentence. Should this acronym be spelled out? 

This document is a form, not a policy, and therefore cannot be analyzed in the 
manner undertaken here. 
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Decision Memo-Centralized Management of Web Servers 
http://inet.ed.gOv/offices/OCIO/it/it4.html 

It appears that the main policy document, U.S. Department of Education World 
Wide Web Server Policy and Procedures, may be enhanced by including a new link at 
page 8 to this document. Decision Memo-Centralized Management of Web Servers. 
Since the language in the main policy document duplicates that of the Decision Memo- 
Centralized Management of Web Servers, it must have originated from this document. 
Providing a link in the main policy document to Decision Memo would provide added 
context for the policy. 

From the statement, “ITIRB requested regular reports from the Deputy Chief 
Information Officer on the implementation of this policy and on the need for any 
modifications” (paragraph 6) it appears that overall enforcement of this policy will be 
conducted by the Information Technology Investment Review Board (ITIRB), although 
the IWG will enforce the exceptions process. 

Two places are given for user feedback to be collected in this document. There is 
the standard mailto link in the footer which solicits questions, comments, and 
suggestions. There is also, however, an explicit statement in the body of the policy, “If 
you have any questions, or require additional information, please contact Gloria Parker, 
at 401-3200” (paragraph 8). This serves to explicitly and specifically invite user 
feedback. 



Exit Icon Disclaimer 

http://www.ed.gOv/offices/OCIO/exit.html 

An inconsistency exists with regard to the Elxit Icon Disclaimer. While the main 
policy document, U.S. Department of Education World Wide Web Server Policy and 
Procedures, refers to the existence of the Exit icon (p. 20, paragraph 3), there is no 
reference either in the main document or at Establishing Links to External Organizations 
(http://www.ed.gov/intemal/xtmlink.html) as to when the text contained at 
<http://www.ed.gOv/offices/OCIO/exit.html> should be used. Specifically, when is this 
disclaimer to be used? 
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Appendix III-2, Implementation Review Checklist 
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Appendix III-2, Implementation Review Checklist 



User Friendliness 

• Majority of pages include no more than 1 V^ screens of text 

• Includes link(s) to main ED site, main ED site program index, principal office site(s), 
organization/program/project site(s), document/collection home page(s) as 
appropriate, depending on level of page in the organization of the site 

• Includes text equivalent for all information contained in graphics; includes ALT text 
for graphics 

• Includes no-ffames alternative to fi'ames 

• Includes alternatives to online interactive forms 

• The links work 

• File format matches the intended usage (used online, offline print or display, offline 
analysis or manipulation); proprietary or unusual formats are explicitly noted; formats 
other than HTML are linked to an HTML page which describes the material in such a 
way that users of site-wide full-text search facilities can find material of interest 

• Files to be downloaded to a user’s machine use HTTP protocol; files larger than 500K 
are compressed in ZIP format and accompanied by an uncompressed ASCII file 
which describes contents and provides instructions for uncompressing; data files are 
accompanied by and ASCII or PDS document which provides adequate 
documentation of the file content and structure 

• Documents larger than 10 screens are divided into multiple, smaller files along 
logical break points. 

• Multifile documents have appropriate table of contents (index.html) page; multifile 
documents of more than 10 files will be available for download as PDF file or ZIP 
file 

• Links to PDF files are accompanied by link(s) to Adobe Acrobat viewer 

• PDF Files larger than 500k are broken down into sections 

• Links to ZIP files are accompanied by link(s) to standard directory containing zip 
utility software and instructions (gopher://gopher.ed.gov/ll/info/utilities/) 

• Links to executable programs are accompanied by appropriate instructions and 
specifications 

• Includes standard navigational buttons where appropriate 

• Visibility — text colors can be read against the overall page background; consecutive 
links are visually separated by dividing character's) 



Other 

• URL naming convention is followed in terms of case (lower/upper) 

• Organization, program, or project sites contain a specific statement describing the 
purpose and content of the document collection. 
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• Pages whose focus is linking to external sites include or link to 1) the text of the 
standard Disclaimer of Endorsement statement and 2) a statement of the scope of the 
collection of external links and the criteria for inclusion. 

• Includes no obvious spelling errors 

• Includes no obviously out-of-date information 

• Includes e-mail address(es)/generic e-mail address(es) where appropriate 

• Icons used for the main ED home page appear to be the standard icon (EDhome.gif) 

• No personal home pages are found 

• Documents include the standard footer [last update date, e-mail address or initials of 
responsible party. . .and link to an appropriate home page (for the document, 
collection, project, program, organization or Department)] 

• Copyrighted material includes notice of reprint permission in appropriate format 
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Appendix III-3, Sample Implementation Review Grid for OESE 



User Friendliness 


URL: 

http://www.ed.gov/offices/OESE/ 


Majority of pages include no more than VA screens 
of text 


0/1 (3+ screens) 


Includes link(s) to main ED site, main ED site 
program index, principal office site(s), 
organization/program/project site(s), 
document/collection home page(s) as appropriate, 
depending on level of page in the organization of 
the site 


1/1 

(but links to programs are 1 or more 
levels down) 


Includes text equivalent for all information 
contained in graphics; includes ALT text for 
graphics (ALT Text can be verified by Doctor 
HTML) 


1/1 


Includes no-fi’ames alternative to frames 


N/A 


Includes alternatives to online interactive forms 


N/A 


The links work (Can be verified by Doctor HTML) 


1/1 


File format matches the intended usage (used 
online, offline print or display, offline analysis or 
manipulation); proprietary or rmusual formats are 
explicitly noted; formats other than HTML are 
linked to an HTML page which describes the 
material in such a way that users of site-wide full- 
text search facilities can find material of interest 


1/1 


Files to be downloaded to a user’s machine use 
HTTP protocol; files larger than 500K are 
compressed in ZIP format and accompanied by an 
uncompressed ASCII file which describes contents 
and provides instructions for uncompressing; data 
files are accompanied by an ASCII or PDF 
document which provides adequate documentation 
of the file content and structure 


N/A 


Documents larger than 10 screens are divided into 
multiple, smaller files along logical break points. 


N/A 


Multifile documents have appropriate table of 
contents (index.html) page; multifile documents of 
more than 10 files will be available for download 
as PDF file or ZIP file 


N/A 


Links to PDF files are accompanied by link(s) to 


N/A 



Adobe Acrobat viewer 




PDF Files larger than 500k are broken down into 
sections 


N/A 


Links to ZIP files are accompanied by link(s) to 
standard directory containing zip utility software 
and instructions 

(gopher://gopher.ed.gov/ll/info/utilities/) 


N/A 


Links to executable programs are accompanied by 
appropriate instmctions and specifications 


N/A 


Includes standard navigational buttons where 
appropriate 


N/A 


Visibility — text colors can be read against the 
overall page background; consecutive links are 
visually separated by dividing character(s) 

Other 


1/1 


URL naming convention is followed 


1/1 


Organization, program, or project sites contain a 
specific statement describing the purpose and 
content of the document collection. 


N/A 


Pages whose focus is linking to external sites 
include or link to 1) the text of the standard 
Disclaimer of Endorsement statement and 2) a 
statement of the scope of the collection of external 
links and the criteria for inclusion. 


N/A 


Includes no obviously out-of-date information 


1/1 


Includes e-mail address(es)/generic e-mail 
address(es) where appropriate 


Yes - initials. Questions &, 
Comments 


Icons used for the main ED home page appear to 
be the standard icon (EDhome.gif) 


0/1 (edhome.gif instead) 


No personal home pages are found 


True 


Documents include the standard footer [last update 
date, e-mail address or initials of responsible 
party. . .and link to an appropriate home page (for 
the document, collection, project, program, 
organization or Department)] 


1/1 


Copyrighted material includes notice of reprint 
permission in appropriate format 


N/A 



Note: The numbers in the right-hand column represent the ratio of the number of pages 
in compliance to the number of pages viewed (# in compliance/# viewed). 
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Appendix III-4, Web Policy Implementation Findings 

Any implementation review checklist items that are not mentioned below were 
either found to be in compliance or were not relevant. 



OESE 

http://www.ed.gov/offices/OESE/ 

Among the sites reviewed for this analysis, the Office for Elementary and 
Secondary Education (OESE) clearly has the largest, most developed site. The 
biographical page for Assistant Secretary Gerald N. Tirozzi 

(http://www.ed.gov/offices/OESE/tirozzi.html) is one of the few that includes a link to 
the office home page. OESE has done a very good job of cross-linking to other sites 
within the Department of Education: 

• http://www.ed.gov/offices/OESE/program.html links to 
http://www.ed.gov/offices/OUS/titlel.html 

• http://www.ed.gov/offices/OESE/CEP/info.html links to 
http://www.ed.gov/offices/OUS/us98.htm 

There were also a number of problems found with certain OESE pages: 

• Many pages are longer than 1.5 screens 

OESE Home page at http://www.ed.gov/offices/OESE/index.html 
Programs and Funding at http://www.ed.gov/offices/OESE/program.html 

- Technical Assistance at http://www.ed.gov/offices/OESE/asst.html 

- Migrant School Locator Demonstration Project at 
http://www.ed.gov/offices/OESE/MEP/PrelimGuide/locator.html 
Office of Migrant Education: Grants at 
http://www.ed.gov/offices/OESE/MEP/grants.html 
Compensatory Education Programs: Grants &. Eligibility at 
http://www.ed.gov/offices/OESE/CEP/grant_eligibilty.html 

• In general the links between pages are good, but problems were foimd with pages 
for Goals 2000 (http://www.ed.gov/G2K/) and Chief of Staff Sarah Lisenby 
(http://www.ed.gov/offices/OESE/lisenby.html) which link back to the ED home 
page, but not to the OESE home page. 

• Other problems with links were found in the form of a lack of integration of the 
fairly new School Improvement Programs web site with the rest of OESE. 
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The School Improvement Programs web site 

(http://www.ed.gov/offices/OESE/SIP/) includes a page for the Magnet 
Schools Assistance Grant Program 
(http://www.ed.gov/offices/OESE/SIP/magnet.html) 

OESE links to a similar page at a different URL 
(http://www.ed.gov/offices/OESE/magnet/msap.html). 

• The ESEA Discussion Forum documents are longer than 10 screens: 

Bilingual Education and Minority Languages Issues at 
http://www.ed.gov/offices/OESE/ESEA/eseobmla.fiil.html 
- Equity Issues at http://www.ed.gov/offices/OESE/ESEA/ eseaqty.fiil.html 
Secondary School Reform at 

http://www.ed.gov/offices/OESE/ESEA/eseahigh.fiil.html 

• The Standards Assessment document 
(http://www.ed.gov/offices/OESE/StandardsAssessment/) consists of more than 
ten files, but does not appear to be available as a PDF or Word document for 
downloading to the user's machine. 

• The "Up" icon from the Standards Assessment document 
(http://www.ed.gov/offices/OESE/StandardsAssessment/) takes the reader back to 
OESE rather than the table of contents for the Standards Assessment document. 

• Comments Requested on Draft Consolidated Performance Report Under 0MB 
Review (http://www.ed.gov/offices/OESE/webpara2.html), found on the What's 
New Page (http://www.ed.gov/offices/OESE/news.html), includes a link to a PDF 
document (http://www.ed.gov/offices/OESE/consol7.pdf) but does not contain a 
link to the Adobe Acrobat reader. 

• The focus of the "Hot Topics" page (http://www.ed.gov/offices/OESE/hot.html) is 
providing external links, but no disclaimer is included or linked from this page. 

• Out-of-date information was foimd in the Events Archive 
(http://www.ed.gov/offices/OESE/events_archive/), where the 1997 meeting 
registration form is still available as a link. 

• A number of the pages reviewed in the OESE site contained no contact 
information in the form of an e-mail address. 

Technical Assistance at http://www.ed.gov/offices/OESE/asst.html 
Technical Assistance: Regional Service Teams at 
http://www.ed.gov/offices/OESE/rsertem.html 
Technical Assistance: Policy Guidance at 
http://www.ed.gov/offices/OESE/policy.html 
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Documents found in OESE include the standard footer, but generally do not 
include mailto links on the initials found. 



Comments Requested on Draft Consolidated Performance Report Under 0MB 

Review at http://www.ed.gov/offices/OESE/webpara2.html 

Magnet Schools Assistance Grant Program for FY 1 998 at 

http://www.ed.gov/offices/OESE/SIP/magnet.html 

Calendar of Events at http://www.ed.gov/offices/OESE/calendar.html 

• The School Improvement Programs page at 

http://www.ed.gov/ofEices/OESE/SIP/programs/ is very poorly designed. 

The OESE web site is very large with a lot of web pages, many of which have 
problems as shown above. However, the OESE site is also the largest site reviewed. 
Though problems were found, there are many more pages which do comply with the 
guidelines. 



Other Findings 

The findings in this section are based on reviews of four additional offices, 
including the Office of Legislative and Congressional Affairs, the Office of the Under 
Secretary, the Office of Management and the Office of Civil Review. 



Office of Legislative and Congressional Affairs (OLCA) 
http://inet.ed.gOv/offices/OLCA/ 

In general, the pages in this web site are compliant, but a few problems were 
noted. Many pages are longer than 1 .5 screens: 

• OLCA Home Page at http://www.ed.gov/offices/OLCA/ 

• Education Legislation at http://www.ed.gov/offices/OLCA/legislation.html 

• Testimony at http://www.ed.gov/ofEices/OLCA/testimony.html 

• Budget Appropriations Bill at http://www.ed.gov/offices/OLCA/budchal .html 

Some problems were noticed with links. Pages that don't link back to OLCA but should 
include: 

• The biographical page for Assistant Secretary Scott Fleming 
(http ://www. ed.gov/ ofEices/ OLCA/ fleming.html) 

• The 5/08/98 letter from Scott Fleming 
(http://www.ed.gov/ofEices/OLCA/eoz2.html) 
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Letters from Secretary Riley that are dated between June 4, 1998 and June 9, 1998 all 
seem to have incorrect document titles (i.e. HTML titles) reflecting the same incorrect 
date of 97/06/16. The title shown for each letter is 97/09/16 - Letter from Secretary 
Riley to Members of the House of Representatives on efforts to block grant education 
funds: 

• http://www.ed.gOv/offices/OLCA/jun041 ltr.html 

• http://www.ed.gOv/offices/OLCA/jun061tr.html 

• http://www.ed.gOv/offices/OLCA/jun081tr.html 

• http://www.ed.gOv/offices/OLCA/jun091tr.html 

It appears that each letter is created by editing a copy of another letter, but the author has 
forgotten to update the document title in each case. All other Implementation Review 
Checklist items were found to be either in compliance or not relevant. 



Office of the Under Secretary (OUS) 
http://www.ed.gov/offices/OUS/ 

In general, the pages in this web site are compliant, but a few problems were 
noted. The majority of pages are longer than 1.5 screens: 

• Summary of the 1999 Budget at 

http://www.ed.gOv/offices/OUS/Budget99/BudgetSum/pages/sum- 1 .html 

• About Us at http://www.ed.gov/offices/OUS/about.html 

• The Federal Budget Process at http://www.ed.gov/offices/OUS/budpro2.html 

• Evaluations of Elementary and Secondary Education Programs at 
http://www.eigov/offices/OUS/eval/elem.html 

Some problems were noticed with links. Pages that don't link back to OUS but should 
include: 

• Asia Pacific Economic Corporation (http://www.ed.gov/offices/OUS/APEC/) 

• FY 1999 Budget (http://www.ed.gov/offices/OUS/Budget99/) 

Documents available in PDF format for which there are no links to the Acrobat reader 
include: 

• The FY 1999 Budget 

(http://www.ed.gov/offices/OUS/Budget99/BudgetSum/index.html) 

• Financial Aid Evaluation (http://www.ed.gov/offices/OUS/eval/finaid.html) 

• Continuous Improvement Management 

(http : //www. ed.gov/ offices/ OUS/ eval/2 1 cent/2 1 stc .html) 

Documents available in zipped format for which there is no link to the zip utility software 
directory include Emergence of Tech Prep at State & Local Levels (1 995) 
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(ftp://ftp.ed.gov/www/ZipDocs/Emerge.zip) which is linked from 

(http://www.ed.gov/pubs/Emergence/). Tliis zip document also uses FTP protocol rather 
than HTTP protocol. Navigational icons appeared to be limited to movement within a 
single web page, such as to the top of the page; all other navigational aids appeared to be 
presented as text-based links. The use of color on the Budget Calendar at 
http://www.ed.gov/offices/OUS/budcal99.html (black on yellow) made the text a little bit 
hard to see, but it wasn't too bad. There is a lot of older information available from the 
OUS site, but all older information is clearly marked as to the publication date. There 
was one instance found where a current page links back to a non-current page: 

• The 1998 Title I Grants page (http://www.ed.gov/offices/OUS/titlel.htm) is 
linked up to the 1997 Title I page (http://www.ed.gov/offices/OUS/us97.htm) 
rather than the current 1998 Title I page 
(http://www.ed.gov/offices/OUS/us98.htm). 

No copyrighted material was found, but there was a notice with the Evaluation Primer 
(http://www.ed.gov/offices/OUS/eval/primerl.html) stating that this was in the public 
domain. 



Office of Management (OM) 

http://www.ed.gov/offices/OM/ 

The components of this office have done the best job of providing disclaimers 
among the sites reviewed. Disclaimers are available on the following pages: 

• The Office of Hearings and Appeals (http://www.ed-oha.org/) has provided a 
link to their disclaimer statement (http://www.ed-oha.org/disclaim.html) right 
on the home page 

• The Family Policy Compliance Office 
(http://www.ed.gov/offices/OM/fpco.html) has included a Disclaimer section 
at the bottom of their home page 

This office has also done the best job of including mission statements among sites 
reviewed. Mission statements can be found on the following home pages: 

• Family Policy Compliance Office (http://www.ed.gov/offices/OM/fpco.html) 

• Human Resources Group (http://www.ed.gov/offices/OM/hrg.html) 

• Labor Relations Group (http://www.ed.gov/offices/OM/wlabor.html) 

• Quality Workplace Group (http://www.ed.gov/offices/OM/wquality.html) 

• Training and Development Group 
(http://www.ed.gov/offices/OM/wtraindg.litml) 

In general the links are good, but a few problems were noted. Documents missing Imks 
back to the ED home page and the OM home page include: 
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• OM: Complaint and Motion (http://www.ed.gov/offices/OM/complain.html, 
http://www.ed.gov/offices/OM/motion.html) does not have links back to 
either OM or ED 

• Letter from Leroy Booker (http://www.ed.gov/offices/OM/hope.html) 

The Collective Bargaining Agreement (http://www.ed.gov/offices/OM/toc.html) is not 
available as a PDF, Microsoft Word or ASCII file for offline display or print; but 
common sense indicates that it should be. Pages offering PDF files for download that 
include a link to the Adobe Acrobat reader include: 

• Family Policy Compliance Office (http://www.ed.gov/offices/OM/fpco.html) 

• Weekly Job (http://www.ed.gov/offices/OM/jobweb.html) 

Some pages include disclaimers: 

• Family Policy Compliance Office at http://www.ed.gov/offices/OM/fpco.html 

• Office of Hearings and Appeals at http://www.ed-oha.org/) 

while others do not 

• Management Systems Improvement at 
http://www.ed.gov/offices/OM/msig.html) 

but in each case external links are not explicitly marked as such. The Weekly Job 
Vacancies (http://www.ed.gov/offices/OM/jobweb.html) appears to be very current. 



Office of Civil Rights (OCR) 
http://www.ed.gov/offices/OCR/ 

OCR includes two major links back to the main ED site, for Search and Year 
2000 information. The Related Links page (http://www.ed.gov/offices/OCR/links.html) 
includes a disclaimer, but does not otherwise mark each external link explicitly. 
However, there are numerous link problems on this site, more than were found on any 
other site reviewed: 

• Pages that link back to the ED home page, but not to the OCR home page 
include biographical pages for -- 

- Norma V. Cantu at http://www.ed.gov/offices/OCR/cantu.html 

- Raymond Pierce at http://www.ed.gov/offices/OCR/pierce.html 

- Arthur Coleman at http://www.ed.gov/offices/OCR/coleman.html 

• Documents that are not linked back to OCR or ED but should be include 
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All 10 of the documents found under the headings “TITLE VI OF THE 

CIVIL RIGHTS ACT 1964 / AFFIRMATIVE ACTION” and “OCR 

REPORTS, STUDIES AND STATISTICS” on the Publications and 

Products page at http://www.ed.gov/offices/OCR/ocrprod.html: 

■ U.S. Department of Education/Office for Civil Rights: Letter of 
Findings to University of California, Berkeley School of Law 
(09/25/92) and settlement agreement. Subject: Considerations for 
determining permissibility of affirmative action admissions program 
under Title VI of the Civil Rights Act of 1964. File Update: 

09/22/97^X. http://www.ed.gov/ofifices/OCR/ucberkle.html 

■ U.S. Department of Education/Office for Civil Rights: Title VI of the 
Civil Rights Act of 1964; Final policy on race-targeted aid (02/23/94). 
Non-discrimination in Federally assisted programs. Replaces the 
proposed policy guidance dated December 10, 1992 and published in 
the Federal Register, at http://www.ed.gov/offices/OCR/racetarg.html 

■ U.S. Department of Education/OJfice for Civil Rights: Letter to 
College and University Counsel confirming policy guidance on race- 
targeted financial aid in light of Podberesky and Adarand decisions. 
(09/07/95). Title VI of the Civil Rights Act of 1964 at 
http://www.ed.gov/offices/OCR/podberes.html 

■ U.S. Department of Education/Ofifice for Civil Rights: Memorandum 
to UCLA. Subject: Title VI of the Civil Rights Act of 1964: 

Investigation of the UCLA undergraduate admissions program 
regarding discrimination against Asian American applicants to the 
College of Letters and Science (L &S) and the School of Engineering 
and Applied Science (SEAS), at 
http://www.ed.gov/offices/OCR/ucla.html 

■ U.S. Department of Education/Office for Civil Rights: Memorandum 
to College and University Counsel (6/30/96). Subject: Title VI of the 
Civil Rights Act of 1964: Consideration of race in making college and 
university admissions decisions and granting financial aid. File 
Update: 07/22/97 at http://www.ed.gov/offices/OCR/dearcol.html 

■ U.S. Department of Education/Office for Civil Rights: Resolution 
Letter to the Washington Legal Foundation (02/21/97), and Florida 
Atlantic University Resolution Agreement . Subject: Race-based 
scholarship and compliance with the Title VI of the Civil Rights Act of 
1964. File Update: 09/22/97 
http://www.ed.gov/offices/OCR/fau.html 

■ U.S. Department of Education/Office of the Secretary: Dear Colleague 
Letter from the Secretary of Education Riley (03/1 9/97). Subject: 
California Civil Rights Initiative, Proposition 209, regarding the 
responsibilities of California's school districts and colleges under 
educational programs and civil rights laws administered by this 
Department. Title VI of the Civil Rights Act of 1964. Affirmative 
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Action. File Update: 09/23/97 at 
http://www.ed.gOv/offices/OCR/prop209.html 

■ U.S. Department of Education/Office for Civil Rights: Memorandum 
to Honorable Rodney Ellis, Texas Senate (4/1 1/97). Subject: Title VI 
of the Civil Rights Act of 1964: Hopwood v. Texas at 
http://www.ed.gov/offices/OCR/hopwood.html 

■ U.S. Department of Education/Office for Civil Rights: Memorandum 
to the Oregon State System of Higher Education (07/03/97), and 
Resolution Agreement. Subject: Tuition Waver Program compliance 
with the Title VI of the Civil Rights Act of 1964. File Update: 

09/22/97 at http://www.ed.gov/offices/OCR/oregon.html 

■ 1998 UNITED STATES DEPARTMENT OF EDUCATION U.S. 
ACCREDITED POSTSECONDARY MINORITY INSTITUTIONS (1) at 
http://www.ed.gov/offices/OCR/minlistl998.html 

Both of the documents found under the subheading “Funding of Athletic 

Scholarships” under the heading ‘TITLE IX OF THE EDUCATION 

AMENDMENTS OF 1972” on the same Publications and Products page 

at http://www.ed.gov/offices/OCR/ocrprod.html 

■ Two letters to Bowling Green University dated July 23, 1998 at 
http://www.ed.gov/offices/OCR/bowlgm.html 

• The Annual Report to Congress for 1997 
(http://www.ed.gov/offices/OCR/AnnRpt97/index.html) includes links to 
OCR National Headquarters and OCR Offices, but these links do not work 
because they are relative links looking for the wrong level in the OCR 
hierarchy of pages. 

• The document US. Department of Education/Office for Civil Rights: 
Summary Sexual Harassment Guidance: Harassment of Students by School 
Employees, Other Students, or Third Parties is broken into two major sections 
(http://www.ed.gOv/offices/OCR/sexharOO.html, 
http://www.ed.gOv/offices/OCR/sexhar01.html) which are linked to each 
other, but nothing else. This document should have a table of contents page 
with a link back to the OCR home page. 
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Appendix III-5, Side By Side Analysis Of Web Policy Documents 
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Appendix III-5, Side By Side Analysis Of Web Policy Documents 

The side-by-side analysis is arranged based on the organization found in the Federal Web 
Consortium Guidelines. This comparison is not a complete review of the differences 
between the Department of Education web policy and the Federal Web Consortium 
Guidelines — it is a limited review of selected topics where the policies appear to differ 
markedly. 
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APPENDIX m-S (continued) 



C3 

Q> 

E 

s 

Q 

O 

Q 

>» 

Q 

"o 

PUi 

A 

Q> 



"cS 

c 

Q> 

■o 



>> 

I 

Q> 

3 

CO 



s 

A 

Q 

s 

■o 

Is) 



C3 

Q> 

E 

u 

e« 

CX 

a> 

Q 



E 

.S 

u 

o 

M 

C3 

o 

U 

A 

Q> 



C9 

U 

Q> 

■a 

Q> 



a 

H 




iO 

C> 







Appendix 44 



•o 

0> 

.S 

‘W 

c 

o 

o 



c 

0> 

E 

3 

u 

o 

Q 

ftj 

"o 

.3 

0> 



NH O 

>< i 

Q 
Z 
(a^ 

Pui 



"rt 

C 

< 

a 

•o 



>> 

.3 

I 

0> 

2 

K 




a:> 



o 

CD 



ERIC 



Appendix 45 



-a 

O) 

3 

.E 

c 

o 

w 



X 

NN 

Q 

Z. 

Pn 

Pn 

< 




VO 

TT 



•O 

§ 

o. 

o. 

< 



O 

CD 



QD 

CD 



er|c 



Side-by-side Analysis of Web Policy Documents 





Appendix 47 



Side-by-side Analysis of Web Policy Documenls 
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APPENDIX ni-5 (continued) 
Side-by-side Analysis of Web Policy Documents 
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Side-by-side Analysis of Web Policy Documents 
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APPENDIX III-5 (continued) 
Side-by-sIde Analysis of Web Policy Documents 
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Side-by-side Analysis of Web Policy Documents 
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Appendix IV- 1, Jeffrey H. Rubin and Ricardo Reimundez, “Log 
Analysis Tools: Site Server is on the Right Track” Network 
Computing, September 15, 1998, pp. 77 - 85 
(http://www.networkcomputing.com/91 7/91 7rl.html). 
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R e view s 



Log-Analysis Tools: Site 
Server is on the Right Track 




with 



Assembling a corporate Web site may 
seem so simple — grab a free copy of a 
Web server, post Just about any content 



and watch all your marketing dreams come true. The reality isn't 



Ricardo quite such a piece of cake, though. Today's Web sites can send 

Reimundez hundreds of megabytes of log files to back-end databases. 



In This Section 



Comparative 

Reviews 

B-Port Remote- 
Access Servers 86 

We tested nine 
models aimed at 
small offices. 




enabling a depth of analysis be- 
yond your wildest imagination. 

Using Web cookies and the 
latest log-analysis programs, you 
can track, store and report on a 
remarkable amount of informa- 
tion about individual visitors to 
your Web site. Tie in your exist- 
ing corporate database, and you 
can customize your online offer- 
ings based on visitors' prefer- 
ences, and eventually generate 
historical reports to find out 
how many are sufficiently satis- 
fied with your site to make re- 



peat visits. With Web servers' 
current configurations and high- 
end log-analysis software, you 
can learn more about a virtual 
user than about a shopper phys- 
ically present in a store. 

To help you determine which 
log-analysis software package will 
best help you realize your site's 
potential, we tested the leading 
contenders at Syracuse University 
in one of Network Computing's 
Real-World Lab^.The good news 
is that we saw vast improvements 
in the products compared with 




Enterprise Log-Analysis Tools 



Feature 


Weight 


MlCflSOfl 
Site Server 3.0 


WebTrends 
Enterprise 
Suite 2.1 


net.Genesis 
net.Anaiysis 
Pro 3.5 


1 Marketwave 
, HitUst 
! Enterprise 4.0 


Marketing-related features 


25% 


4.5 


4 


4 


3.5 


Technical-related features 


25% 


3 


3.5 


4.5 


1 4 . ' 


Performance 


25% 


5 


4 


3.5 


2 


Ease of use 


15% 


4 


5 


2 


4 


Price 


10% 


5 


4.5 


3 


2 


Total score 


4.23 


4.08 


3.60 


3.18 


te4.3,B^.5,(^.5,D>1.5,F<1.5 

A-C grades indude + or - in their ranges 




■ i+ . 







Total scores and weighted scores are based on a scale of 0-5. 

Customize the results of this Report Card to your environment using the tatwaethfo Report Cant, a Java applet on Ketwork Computimg 
Online , at www.networkeompuUng. com. 

^ Products that score a B or above in our tests receive the Network Computing Flying Colors Award. 



versions we tested a year ago. All 
the products can produce sophis- 
ticated, customizable reports that 
will deliver valuable information 
to Web administrators and senior 
management alike. Of course, no 
matter how rich a report you get, 
remember that software is still 
just a tool; someone in the orga- 
nization must be equipped to an- 
alyze the reports and apply the 
information to improve the site. 

And even with the incredi- 
ble advances in log-analysis 
software made during the past 
year, most packages still could 
do a better job of error analysis. 
net.Genesis Corp.'s net.Analysis 
is the only product we tested 
with built-in, on-the-fly reports 
featuring intelligent error 
checking. The desire for true 
user demographics also remains 
unfulfilled. Although most log- 
analysis vendors claim to pro- 
vide detailed user demographics 
(including country, state and 
city), they don't make it clear 
to Web administrators that the 
information is coming from 
Whols, an Internet service that 
reports on contact information 
for DNS and IP Internet do- 
mains registered by the Inter- 
NIC. Whols queries are not a 
reliable source of demographic 
data because those who reach 
your site through a major ISP 
will appear by default to be lo- 
cated at the service provider's 
i headquarters. For example, ac- 
i cording to Whols queries, all 
I America Online users appear to 
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9 







fleVlBVIsEnterprise Log-Analysis Tools 



be at the online service provider's 
headquarters in Vienna, Va. However, 
none of the packages write these loca- 
tions into their reports. 

Getting Down to Business We separated 
each product's reporting features into 
two categories— marketing and techni- 
cal— to reflect the different needs of 
the people accessing log-analysis data. 
While Web administrators focus on 
page layout, errors, path analysis and 
browser summaries, marketers eye ad 
analysis, user demographics and 
search-engine queries (the ability to 
pick out keywords in user searches). 

We evaluated commercial enter- 
prise-class Web server log-analysis 
tools from Marketwave Corp., Mi- 
crosoft Corp., net. Genesis and Web- 
Trends Corp. We judged the four con- 
tenders based on performance (how 
long it took to import and run reports), 
ease of use, quality of reports, infor- 
mation provided to administrators and 
marketers, and price. 

Microsoft's Site Server 3.0 garnered 
our Editor's Choice award; it clearly 
provided the highest level of perfor- 
mance and functionality at the lowest 
cost. Site Server is capable of full ad- 
vertising and user analysis at levels 
well beyond those of its rivals. Site 
Server can read nearly any log-file for- 
mat and very efficiently move infor- 
mation to a SQL database. 

Our second choice is WebTrends 
Enterprise Suite 2.1; while it doesn't 
match Site Server in terms of function- 
ality, it compensates with its ease of 
use. Enterprise Suite can create com- 
prehensive visit-analysis reports, and 
also does a powerful job highlighting 
keywords from users' search queries. 
Enterprise Suite is a great value for a 
company that doesn't need the depth 
of the Site Server package. 

Microsoft Corp. 

Site Server 3.0 

Starting at $1,239, Microsoft's Site 
Server 3.0 gives Web administrators 
the most functionality for the lowest 



cost. Our testing focused on Usage An- 
alyzer, which is just one component of 
the Site Server package. Even with it's 
economical price. Site Server seems to 
handle all the log-analysis functions 
any other package offers. Microsoft also 
includes knowledge management and 
publishing features as part of the regu- 
lar p°ackage, and transactional analysis 
in the Commerce Edition. (We did not 
test these features for this article.) 

At first, we were somewhat over- 
whelmed by Site Server's configuration 
and management features. Site Server 
3.0 uses Microsoft's wizards from instal- 
lation through configuration. The most 
difficult thing to adapt to was the inclu- 
sion of a separate database wizard to set 




we put the package at the head of the 
pack. When you request a report to 
determine who is looking at the Web 
site, most products query an internal 
database that contains InterNlC 
Whols information. While 
this speeds perfor- 
mance you may 

find that IP ad- q„q|q£ 

dresses registered 

after the product was shipped cannot 
be linked to an organization. 

To avoid this, Site Server queries 
InterNlC directly for Whols informa- 
tion. While this slows response the 
first time the software runs, the infor- 
mation is cached for future use, allow- 
ing for quicker imports on subsequent 




Creating custom reports in Site Server is a simple drag-and-drop operation. 



up the database, import wizard to set 
up the import and report writer to run 
a report. Even though it makes sense to 
break up the functions with individual 
tools, Microsoft is the only vendor that 
does so. Site Server runs exclusively on 
I Windows NT, as do two of the other 
i three packages we tested; net.Analysis 
I runs on Unix as well as NT. 
i The accuracy of Site Server's orga- 
1 nizational lookups is one key reason 



requests. (The cache expires on a regu- 
lar basis and can be configured by the 
administrator.) The distinct advantage 
of querying InterNlC is that the orga- 
nizational information is updated reg- 
ularly, making reports more accurate. 
No other product has a mechanism in 
place to update the organizational 
database that ships with the produa, 
aside from regularly scheduled, sepa- 
rately priced updates. 
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Enterprise Log-Analysis Tool Features 





Marketwave 
Hit list 
Enterprise 4.0 


Microsoft 

Site 

Server 3.0 


net.Genesis 
netAnalysis 
Pro 3.5 


WebTrends 
Enterprise 
Suite 2.1 


Analysis: 


Supports distributed servers 
(mirrored or split content) 


• 


• 


• o (Can read them 

as wild cards) 


Supports multipte log formats 


• 


• 


• 


• 


Documentation sources 


Online 


Online User guide, online User guide, online 


Proxy server log support 


• 


• 


• 


• 


Reverse ONS lookup 


• 


• 


• 




Query string analysis (internal) 


• 




• 


Keywords only ' 


Query string analysis (external 
. search engines) 


• ' 


• 


• 


Keywords only 


Customized filtering 


• 


• 


• 


• 


Advertising analysis. . 


• (Using 
redirca script) 




• 


• 


Rexibility of custom reports 


• 


• 


• 




Pathanalysis 


• 


• 


• 


• 


DNS caching 


• 


• 


• 


Not within SQL 


Cookie analysis 


• 


• 


• (Proprietary 
plug-in) 


- • 


Compressed log support 


• (grip and rip) 


• (grip and rip) 


• (grip and rip) 


• (grip and rip) 


Reporting: 


Calculates visits/visitors 


• 


• 


• 


• 


Import and report progress 
status indicator 


• 


• 


• 


. • 


Report output formats 


HTML Word. 
ASCn. comma > 
deliminated file 


HTML 
Word. Excel 
FTP, e-mail 


HTML Word. 
Excel text 
comma- 
deliminated file, 
FTP. e-mail 


HTML Word 
Excel texi. 
comma - 
deliminated file. 
FTP. e-mail 


Wizard installation 


• 


• 


• 


• 


Domain breakdown 


• 


• 


• 


• 


Geographic breakdown 


• 


• 


• 


• 


Advanced error checking 


o 


o 


• 


o 


Advanced referrer checking 


o 


• 


o 


• 


Browser breakdown 


• 


• 


• 


• 


Bandwidth summary 


• 


• 


• 


• 


Comparison to previous 
logs/tiends over time 


• 


• 


• 


• 


Management 


Remote administration 


• 


• 


• 


• (NT only) 


. Scheduling for imports 
and reporting 


l-minute 
increments, 
hourly, 
daily, weekly, 
monthly, yearly 


Hourly, daily, 
weekly 


Daily, weekly, 
monthly, 
quanerly 


S-minute 
inaemems. 
hourly, daily, 
monthly, yearly 


Database Support 


Built-in DBMS 


• (Proprietary) 


• (Access) 


• (Unix comes 
with Informix) 


• (Proprietary) 


Attach to DBMS (ODBC) 


• (SQLServer 
or Orade7/8) 


• (SQLServer) 


• (SQL Server) 


• (SQL Server 
or Orade7/8) 


Platform Support 


Windows95/98 •; 


• 


o 


o 


• 


v\ Windows NT 


• 


• 


• 


• 


y Unix 


o 


o 


Solaris 2.5 or 
2.6.AIX4.1.X 


o 


-: ;Ycv 2000-compliant . 


• 


• 


Unknown 


• 



• Y« O NO 



Site Server includes an Access data- 
base run-time engine, but we chose to 
use SQL Server as our database for re- 
pons based on historical data. After we 
installed the latest ODBC (Open Data- 
base Connectivity) drivers. Site Server 
led us through the SQL Database setup, 
even determining how large our data 
store should be based on the size of our 
log file. Although this seems elemen- 
tary, don't take it for granted: We had 
to run Marketwave's Hit List four times 
before we created a data store large 
enough to support our import. Once 
Site Server was installed, it was a snap 
to run reports, set up filters to improve 
performance and focus on any point of 
interest in the logs. The filter setup is 
included within the report wizard. 

All the enterprise-level products we 
tested can be connected to an ODBC- 
compliant database. Importing log files 
into one of these databases gives admin- 
istrators several performance advantages 
and allows them to tie log information 
to other customer information in their 
databases. For a more detailed explana- 
tion of the differences between enter- 
prise-level produas and standard log- 
analysis tools, see 'Enterprise Solutions 
vs. Workgroup Solutions,' on page 82. 

Site Server's reports go beyond the 
scope of any competitor's. Most log- 
analysis tools break out domain infor- 
mation into the top six U.S. domain 
names and then international informa- 
tion. You may see that the Web site had 
X number of requests from .com do- 
mains. X number from .org domains and 
so on. Site Server reports on all this and 
adds ISPs to the mix; it also details the 
leading requests from each top-level do- 
main. including Canada, Canadian ISPs 
and other international users. 

We were slightly disappointed that 
1 Site Server does not offer intelligent 
I error reporting. Web administrators 
I need to see what errors users received 
I while at the Web site, identify the pages 
j that contain those errors and possibly 
1 learn what browsers the visitors were 

I 

1 using when the errors were received. 

1 Site Server edged out net.Analysis in 
1 import speed. It took approximately 31 
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minutes; by comparison, Hit List 
needed 3 hours and 14 minutes. After 
all importing is done, Site Server shows 
the exaa length of import time, which 
could prove useful when a network ad- 
ministrator is making up a schedule. 
Site Server also generates MS Word re- 
ports right before your eyes. 

WebTrends Enterprise Suite 
Version 2.1 

WebTrends Enterprise Suite's biggest 
asset is its ease of use. From installa- 
tion through report generation, the 
software makes log analysis look sim- 
ple. Its interface isn't quite as over- 
whelming as Site Server's, and it may 
be a better bet for Web sites that don't 
have a need for all of Site Server's 
bells and whistles. 

Enterprise Suite performs well 
across the board, but one or two miss- 
ing features keep it a few steps behind 
Site Server. First, it lacks support for 
multiple physical servers. With many 
enterprise Web sites now employing 




You can customize Enterprise Suite*s reports to match the 
look of your other business materials. 



several Web servers 
within the organiza- 
tion (for example, mar- 
keting.company.com and 
sales.company.com), it is 
increasingly important 
to track a user's visit 
across different Web 
servers. Site Server, 
net. Analysis and Hit 
List are all up to the 
task; Enterprise Suite 
merely allows a user to 
stitch together log files 
for analysis. 

Alternatively, En- 
terprise Suite sup- 
ports multiple Web sites reporting to 
one log file. If, for example, an ISP 
has multiple Web sites running on 
the same physical Web server. Enter- 
prise Suite can differentiate between 
them and create customized reports 
for each site. The software's schedul- 
ing features let you create those re- 
ports and FTP them to your clients' 



accounts automatically. 

Enterprise Suite includes ODBC 
connectivity along with WebTrends' 
proprietary FastTrends database. 
There's just one problem with the 
implementation: The only incentive 
to use FastTrends is its ability to track 
trends over time. Unlike the other 
products we tested, WebTrends does 



Enterprise Solutions vs. Workgroup Solutions 



M 

A MO matter how you slice It, performing thou- 
sands of DNS lookups and hundreds of title look- 
ups takes time. But at least when you log Into a 
database, the Information Is stored for future use; 
by the second or third time you run your software, 
you'll see significant performance gains. 

But let's face facts: Many Web sites do not re- 
ceive millions of hits per day. Though you may be 
looking for an easier way to analyze your log traf- 
fic, a smaller operation may lack the computing 
power or software (specifically a relational data- 
base) to benefit from an enterprise solution. If so, 
y.’ . 

workgroup software such as Aquas' Bazaar Ana- 
lyzer Pro, Netrics.com's SurfReport, Sane Solu- 
tions' NetTracker and WebManage Technologies' 
Netintellect may fit your needs, / 



These workgroup solutions have one Important 
drawback compared with the enterprise log- 
analysis tools we tested: While they adequately 
analyze Web traffic and produce reports, they do 
not allow you to Import your log files Into an 
ODBC (Open Database Connectlvlty)-compllant 
back-end database, ^ — r' - 

Using an ODBC database enables you to pro- 
duce historical reports, which could help Identify 
trends over time and provide a clearer picture of 
Web-sIte usage (such as keeping an ongoing 
record of the most frequently used keywords or a 
list of all the sites that link to your site). Rnally, 
logging Into an ODBC database lets you combine 
your log profiles with any other customer Informa- I 
tion you may have In a separate database. | 
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Analyzing the Logs: How We Tested 



tested all products on a Dell Computer Corp. Optiplex Pen- 
tium II running at 300 MHz with 256 MB of RAM under the Windows 
NT Server 4.0 operating system. Ail logs were imported into 
^ Microsoft Corp.’s SQL Server 6.5 database. 

The log flies we tested ranged In size from 12 MB to a 1.2-GB 
proxy log (although none of the products could handle a file of this 
size). We obtained two of our log flies from live corporate sites: one 
from a Fortune 100 company and another from a worldwide stan- 
dards organization. For benchmark testing, we used a 100-MB log 



Time (Basic) 



Log Analysis Performance 



Marketwave Microsoft net.6enesis 

Hit List Site Server net.Analysis 

Enterprise 4.0 3.0 Pro 3.5 




3hrs, 31 min. 49 min. 

14 min. 



Time (Full Lookups) 




lihrs, 
15 min. 




2hrs, 
23 min. 




WebTrends 
Enterprise 
Suite 2.1 




2hrs, 
36 min. 




3hrs, 
46 min. 



file that consisted of million entries (more than 1 million hits) — 
a size indicative of what a large company might deal with If it vvere 
to rotate its log files on a daily basis. 

We tested each product under two performance benchmarks, in 
both performance tests, each product imported the 100-MB log file 
into the SQL database. The first time through, we asked each prod- 
uct to Import the log file, but not to look up page titles or DNS 
lookups. (We called this our basic import.) After clearing ail data- 
bases, we imported the log file again, this time performing title and 
DNS lookups. (This was our full import.) 

Page title lookups require the software to contact the Web server 
and retrieve the page titles from the actual HTML pages. DNS lookups 
take each IP address and attempt to resolve the domain name. 
Although title and DNS lookups can yield valuable Information, as you 
can see from the results of the two performance tests we conducted. It 
may take twice as long, or even longer, to run these imports. 



not cache the IP addresses and title 
lookups. On top of that, using SQL 
takes a toll on performance. It took 2 
hours and 36 minutes to import a 100- 
MB log file into a SQL database with- 
out doing DNS and title lookups; the 
same log file under the same configura- 
tion took just eight minutes to analyze 
when a SQL database was not in use. 

Enterprise Suite impressively breaks 
down every major search engine and re- 
ports on the top query strings from 
each. Enterprise Suite's reports are 
straightforward and comprehensive, and 
the software uses no additional filters. 

Ease of use and fast analysis (with- 
out logging to a database) make En- 
terprise Suite well worth its $1,499 
price tag. If you're looking for a soft- 
ware package to create quick, high- 
quality reports on Web- server aaivity, 
this one's for you. 

net.Genesis Corp. 
net.Analysis Pro 3.5 

net.Genesis' net.Analysis Pro 3.5 offers 
very robust filters, including some 
aimed squarely at the enterprise mar- 
ket, but the package carries a hefty 
price tag. The NT version lists for 
$4,495, and the Unix version (for So- 
laris 2.5 or 2.6 or AIX 4.1.x) is tagged 
at a whopping $7,495 (including an 
Informix database). Then again, 
net.Analysis is the only package we 
tested that even has a Unix release. 

We found net.Analysis' Web-based 
interface annoying. For example, to 
change a configuration option, you 
have to make the change and then 
click on an "OK" button every time. 
When you do, a new saeen loads and 
asks if you want to save your changes. 
At first, this may seem comforting, but* 
after clicking on 10 separate configura- 
tion options, the redundancy began to 
get on our nerves. Further redundancy 
hit during import, when the software 
asks a series of questions pertaining to 
the location and type of your log files. 
There were times when we were an- 
swering the same question twice. 

net.Analysis produced the best tech- 
nical reports of the software packages 
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we tested, and it is the only package to 
offer built-in reports that incorporate 
intelligent error checking. Hit List and 
Enterprise Suite report on the status 
code of errors, while Site Server can re- 
port only on errors using advanced fil- 
ters. net.Analysis reports a status code, 
summary of errors and the pages on 
which users encountered the errors. 

Like Site Server, net.Analysis has a 
wizard-based installation for the SQL 
Server, and it is intelligent enough to 
know how large the data store should 
be based on the size of your log files. 
Unfortunately, it seemed to take 
longer for net.Analysis to create the 
data store than if we had configured 
SQL Server manually. 

net.Analysis crunched away at a 
100-MB log file (without DNS and 
title lookups while importing into 
SQL Server) in 49 minutes— only 
Site Server was faster. However, 
when we tried to do DNS and title 
lookups into SQL Server, the soft- 
ware hung on us twice. Both times it 
ran fine for the first four hours, and 
then our CPU utilization skyrocketed 
to 100 percent for hours. Ultimately, 
net.Analysis failed at this test, and the 
problems that resulted kept us from 
getting accurate readings for our per- 
formance worksheet. net.Genesis was 
unable to provide a fix by the time we 
went to press. 

Perhaps in hopes of offsetting 
net.Analysis' clunky interface and com- 
plex setup, the vendor has a special ser- 
vice called FastStart aimed at enterprise 
users. For $2,000 a day for a two-day 
program, excluding travel expenses, 
net.Genesis will send a technician to de- 
liver the produa, install and configure 
it, run several imports and provide end- 
user training. Given the complexity of 
net.Analysis, it may be more of a neces- 
sity than a luxury. 

Marketwave Corp. 

Hit List Enterprise 4.0 

Marketwave has quite a bit of work 
ahead of it before Hit List can truly 
compete with the other ODBC-com- 
pliant log-analysis tools we tested. 

O 




Although Hit List worked with our 
SQL Server, setup was no picnic. Site 
Server, net.Analysis and Enterprise 
Suite all needed a data store roughly 
three times the size of the log file we 
wanted to import. Hit List did not 



have a database wizard, so we went 
ahead and created a 300-MB data 
store. When that didn't work, we 
created a 400-MB data store— no 
luck there either. Finally, at about 
600 MB, the import worked. In con- 
versations with Marketwave, we 
learned that Hit List isn't optimized 
for ODBC-compliant databases, and 
that ODBC connectivity is scheduled 
to be one of the key improvements 
in the next release. 

While Hit List does create ade- 



quate reports, there is nothing within 
them that gives the product an edge 
on its competitors. For instance. Hit 
List can aeate advertising reports, but 
it will only count the clickthroughs if 
you first redirect them through a 
Marketwave script. 
Site Server, Enterprise 
Suite and net.Analysis 
all can create full ad- 
vertising analysis with 
relative ease. 

And the size of Hit 
List's database isn't its 
only problem. Import- 
ing the lOO-MB log 
file into SQL Server 
and performing title 
and DNS lookups took 
more than 1 1 hours, 
or nearly eight hours 
longer than Enterprise 
Suite (and this doesn't 
take into account that 
net.Analysis crashed after four 
hours). Finally, at nearly $7,000, Hit 
List's pricing is out of the ballpark. ■ 

Jeffrey H. Rubin is an adjunct professor 
with the School of Information Studies at 
Syracuse University, and a consultant for 
Internet Consulting Services. Ricardo 
Reimundei is an independent contractor 
based in Syracuse, N.Y. Send your com- 
ments on this article to Rubin at 
jhrubin@istweb.syr.edu or Reimundez at 
rica rdo@reim undez.com. 




The Language Manager in Hit List Enterprise makes it 
easy to translate terms into different languages or non- 
technical language that makes sense to all users. 



Uendor Information 



Hit List &iterprise 4.0, $6,995, 
Marketwave Corp., (800) 521-8176, 
(206) 682-6801; fax (206) 682-6805. 
info@marketmve.com or 
www.marketwave.com 

neLAnalysis Pro 3.5, $7,495, (So- 
laris), $4,495 (Windows NT), neLGene- 
sis Corp., (617) 577-9800; fax (617) • 
577-9850. sales@netgen.com or 
www.netgen.com 

Site Server 3.0, $1,239 (one ' 
server/five clients), $2,109 (one 




server/25 clients), Microsoft Corp., 
(800) 426-9400, (425) 882-8080; 
fax (425) 936-7329. 
www.microsoft.com/siteserver 

WebTrends Enterprise Suite ver- 
sion 2.1, $1,499, WebTrends Corp., 
(888) WEBTRENDS, (503) 294-7025: 
fax (503) 294-7130. saies@web- 
trends.com or www.webtrends.com 

For more information on products test- 
ed in this issue, go to 
www.networkcomputing.com/iinks. 
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Appendix FV-2, Department Path Report 

First date analyzed: 9/20/98 
Last date analyzed: 9/26/98 

Lists the top 5 first requests, then the top 10 second requests for each of the top first requests, then the top 

10 third requests for each of the second requests. 



/offices/OPE/Professionals/ 



/funding.html 



/pubs/ 



/stats.html 



/EdRes/ 



Subtotal (/flnaid.html) 



/pubs/ 



/pubs/edpubs.html 



/pubs/pubdb .html 



/ncwsletters.html 



/pubs/parents.html 



/pubs/legsregs.html 



/pubs/studies.html 



/BASISDB/EDPUB/search/SDF 



/pubs/resdir.html 



First request V 

Secund request 
Third request 


# of visits 


/ 




/ 




/ 


574 


/free/ 


10 


/flnaid.html 


9 


/inits.html 


9 


/about.html 


8 


/databases/ERIC_Digests/index/ 


8 


/stats.html 


8 

1 


/news, html 


7 


/pubs/EDInitiatives/98/98-09-1 1 .... 


7 1 


/topicsaz.html 


7 1 


Subtotal (/) 


647 


/finaid.html 




/offices/OPE/express.html 


66 


/DircctLoan/ 


61 


/prog_info/SF A/StudentGuide/ 


34 


/inits/hope/ 


31 


/offices/OPE/ 


26 



16 



ll 



ll 



284 



36 



32 



25 



25 



22 



17 



12 



12 
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1 /pubs/collect.html 




1 /pubs/ 


9] 


1 Subtotal (/pubs/) 


200 j 


1 /stats.html 




1 /stats.html 


22 1 


1 /topicsaz.html 


22 1 


1 /prog info/ERIC/ 


2 \\ 


1 /pubs/ 


20 1 


1 /pubs/studies, html 


16 1 


1 /EdRes/ 


15 1 


1 /databases/ERlC_Digests/index/ 


13^ 


1 /NLE/ 


12 1 


1 / 


9 1 


I /offices/OERI/ 


9 1 


1 Subtotal (/stats.html) 


159 1 


1 /programs.html 




1 /Welcome/overview.html 


60^ 


1 /BASISDB/EROD/direct/SF 


3T| 


1 /pubs/Iegsregs.html 


21 1 


1 /EdRes/ 


18 I 


1 /topicsaz.html 


16 1 


1 /about.html 


13 I 


1 /pubs/ 


13^ 


1 /stats.html 


13 1 


1 / 


12 1 


1 /funding.html 


10 1 


1 Subtotal (/programs.html) 


207 1 


1 /EdRes/ 


n 


1 /Programs/bastmp/S E A. htm 


33 1 


1 /EdRes/EdFed/OtherED.html 


20 1 


1 /EdRes/EdFed/ERIC.html 


19 1 


1 /EdRes/EdSupport.html 


i7] 


1 /topicsaz.html 


17^ 


1 /EdRes/EdCurnc.htmI 


12 1 


1 /programs.html 


12 1 


1 / 


11 1 


1 /EdRes/EdAssoc.htmI 


10 1 


1 /stats.html 


9 1 


1 Subtotal (/EdRes/) 


160 I 


1 /free/ 


1 


1 /free/subjecthtml 


97 1 


1 /free/whathtml 


70 1 
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/free/sitemap.html | 


22 1 


/funding.html | 


11 1 


/free/kids.html | 


6 1 


/free/s-lanart.html | 




/ 1 


sj 


/databascs/ERIC_Digests/index/ 1 


5 1 


/free/s-edtech.html j 


5 1 


/flnaid.html | 


4 ] 


Subtotal (/free/) I 


231 1 


/fundmg.html | 


1 


/offices/OPE/Funding/ | 




/legislation/FedRegister/ | 


40^ 


/flnaid.html | 


32 I 


/pubs/KnowAbtGrants/ | 


22 1 


/ofFices/OBEMLA/funding.html 


14 { 


/legislation/FedRegister/announc... | 


13^ 


/funding.html | 


12^ 


/ 1 


9 ] 


/abouthtmi | 


y\ 


/news.html | 


1 1 


Subtotal (/funding.html) | 


1 


/topicsaz.html | 




/topicsaz/ 


1 1 


/topicsaz/topicsqz.html 


1 


/topicsaz/topicsip.html 


1 1 


/EdRes/ 


I M 


/ 


1 ^ 1 


/topicsaz.htmi 


1 4 ^ 


/offices/OERI/BlueRibbonSchools/ 


1 M 


/officcs/OM/edjobs.html 


1 


/programs-html 


1 ^ 1 


/dirs.html 


1 ^ I 


Subtotal (/topicsaz.html) 


1 213^ 


/pubs/EDInitiatives/98/984l9-l l.html 


1 


/pubs/bbecho9 8/ 


26^ 


/PressReleascs/08-1998/wh-083 1 .html 


20 ] 


/inits.html 


13 1 


/pubs/TeachersLead/ 


13 1 


/stats.html 


13^ 


/funding.html 


l7] 


/pubs/EDInitiatives/ 


12 1 


/news.html 


® 1 
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1 /pubs/EDInitiatives/98/98-02-06.... 


8 1 


/pubs/EDInitiatives/98/98-08-28.... 


8 1 


1 Subtotal (/pubs/EDInitiatives/98/98-0...) 


137] 


1 Subtotal (/) 


2,431 1 


1 /ofrices/OPE/Students/ 




1 /offices/OPE/Students/apply.btinl 


1 


1 /offices/OPE/express.html 


215 1 


1 /BASISDB/TITLE4/search/sf 


27 j 


1 /offices/OPE/Students/ 


12 I 


1 /PDFDocs/98fafsafinal/fonn.pdf 


10 1 


1 /prog info/SF A/F AFS A/ 


8 1 


1 /BASISDB/nXLEIV/search/SDF 


5 1 


1 /offices/OPE/Students/apply.html 


4 1 


1 /offices/OPE/Students/apply/fexp... 


3 1 


1 /offices/OPE/Students/student.html 


3 1 


1 / 


2 I 


1 Subtotal (/oflices/OPE/Students/apply...) 


287| 


1 /BASISDB/TITLE4/search/sf 




1 /BASISDB/nXLEIV/search/SDF 


232^ 


1 /BASISDB/nXLEIV/search/DDW 


20 1 


1 /BASISDB/nXLEIV/search/SF 


11 ] 


1 /BASISDB/TITLE4/search/sf 


10 1 


1 /offices/OPE/express.html 


8 1 


1 /offices/OPE/Students/apply.html 


6 1 


1 /offices/OPE/Students/ 


3 1 


1 /BASISV8/dnload/schoolcode.zip 




1 /offices/OPE/Students/fedaid.html 


2 1 


1 Subtotal (/BASISDB/TITL£4/search/sf) 


294 1 


1 /offices/OPE/Students/studenthtml 




1 /progJnfo/SFA/FYE/ 


55 1 


1 /prog_info/SFA/LSA/ 


29 1 


1 /prog info/SF A/StudentGuide/ 


25 I 


1 /DirectLoan/ 


16 1 


1 /offices/OPE/Students/ 


13 1 


1 /offices/OPE/Students/other.html 


12| 


1 /offices/OPE/Students/apply.html 


11 1 


1 /offices/OPE/express.html 


7 1 


I /prog info/SFA/StudentGuide/1 998^9/ 


M 


1 /officcs/OPE/Partners/ 


4 1 


1 Subtotal (/ofTices/OPE/Students/stude...) 


178 1 


1 /ofTices/OPE/Students/ 


1 


1 /offices/OPE/Students/apply.html 


20 1 
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/offices/OPE/Students/student.html 


12 1 


/BASlSDB/TITLE4/search/sf 


9| 


/offices/OPE/Professionals/ 


6 1 


/offices/OPE/express.html 


5 1 


/o ffices/OP E/S tudcnts/ 


5 1 


/ 


4 1 


/BASISDB/TITLEIV/search/SDF 


3 1 


/ofTices/OPE/Paitners/ 


T| 


/ofiices/OPE/Sludents/fedaid.html 


3 I 


Subtotal (/offlces/OPE/Students/) 


7T| 


/BASlSDB/TITLEIV/search/SDF 


1 


/BASISDB/TITLEIV/search/SDF 


3F| 


/BASISDB/TITLEIV/search/DDW 


18 1 


/BASISDB/TITLEIV/search/SF 


iT] 


/offlces/OPE/Students/fedaid.html 


3 1 


/offices/OPE/Students/apply.html 


T| 


Subtotal (/BASISDB/TITLErV/search/SDF) 


74 I 


/offlces/OPE/Students/fedaid.html 


1 


/officcs/OPE/Students/ 


11 1 


/officcs/OPE/S tudents/apply. html 




/offices/OPE/Students/student.html 




/officcs/OPE/Partners/ 




/offlccs/OPE/S tudents/help. html 


4 1 


/offlccs/OPE/DCS/ 




/offlces/OPE/Professionals/ 


2^ 


/offlces/OPE/Students/fedaid.html 


T| 


/offlces/OPE/Students/hopegd.html 


T] 


/prog_info/SFA/FYE/ 


2 1 


Subtotal (/ofTices/OPE/Students/fedai...) 


50^ 


/offices/OPE/express.html 


1 


/ofiices/OPE/Students/apply/fexp... 




/PDFDocs/98fafsafinal/form.pdf 


9 1 


/BASISDB/TITLE4/search/sf 


5 1 


/offices/OPE/express.html 




/offices/OPE/Students/apply.html 


5 1 


/offices/OPE/Students/ 


4 1 


/offices/OPE/Students/fedaid.html 


2 1 


Subtotal (/ofrices/OPE/express.html) 


39 1 


/ofncrs/OPE/Students/hopegd.html 


1 


/offices/OPE/Students/student.html 


11 1 


/offices/OPE/S tudents/apply . html 


8 1 


/officcs/OPE/Students/ 


7 1 
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/offices/OPE/Students/fedaid.html 


AJ 




/offices/OPE/Students/help.html 


tA 




/offices/OPE/Students/hopegd.html 






Subtotal (/offices/OPE/Students/hopeg...) 


32^ 




/ofrices/OP£/Students/help.htinl 






/ofEices/OP E/Students/ 


8 1 




/offlces/OPE/Students/apply.html 


3 1 




/offices/OPE/Students/studenthtml 


3 ] 




/offices/OPE/exprcss.html 


2 




/offices/OPE/Partners/ 


2 




/office s/OPE/Professionals/ 


2 




/offices/OPE/Students/fedaid.html 


2 




Subtotal (/offices/OPE/Students/help....) 


22 




/ 






/offices/OPE/Students/apply.html 


5 




/finaid.html 


4 




/BASISDB/TITLE4/search/sf 


3 1 


/ 




/offices/OPE/express.html 


ll 


/offices/OPE/Students/ 


2 1 


/offices/OPE/Students/studenthtml 


2 1 


/prog^info/SF A/StudentGuide/ 


2 ] 


Subtotal (/) 


2 T] 


Subtotal (/offices/OPE/Students/) 


1,070 1 


/prog info/SFA/StudentGuide/ 




/prog info/SFA/StudentGuide/ 1998-9/ 




/prog_info/SF A/StudentGuide/ 1998... 


293 1 


/prog_info/SFA/StudentGuide/l998... 


32 1 


/prog info/SFA/StudentGuide/ 1998... 


23 I 


/prog_info/SFA/StudentGuide/ 1 998 ... 


18 I 


/offices/OPE/express.html 


15^ 


/prog info/SFA/StudentGuide/ 1998-9/ 


15 1 


/prog_info/SFA/StudentGuide/ 1 998. .. 


ll 1 


/PDFDocs/StdGde98.pdf 


io| 


/offices/OPE/Students/student.html 


9 1 


/prog^in fo/S F A/StudentGuide/ 1 998 . . . 


9j 


Subtotal (/prog_info/SFA/StudentGuide...) 


435^ 


/offices/OPE/expres&html 




/offices/OPE/Students/ 


^ 1 


/PDFDocs/9 8 fafsafinal/fonn.pdf 


21 1 


/offices/OPE/Students/apply/fexp... 


14^ 


/prog_info/SFA/StudentGuide/ 1 998-9/ 


8 I 



/offices/OPE/express.html | 


7 j 


/offices/OPE/Students/apply/dwnl... 1 


7 J 


/PDFDocs/98fafsafinal/inst.pdf | 


7^ 


/ 1 


iJ 


/offices/OPE/home.html | 




/offices/OPE/Studenis/apply.html | 


2 1 


Subtotal (/officcs/OPE/express.htnil) | 


97\ 


/prog_info/SFA/StudentGuide/1998-9/S... | | 


/prog info/SFA/StudentGuide/1998... | 


19 1 


/prog info/SFA/StudentGuide/ 1998-9/ | 


12^ 


/prog info/SFA/StudentGuide/ 1998... | 


9| 


/prog_info/SF A/StudentGuide/ 1 998 ... | 


l\ 


/prog info/SFA/StudentGuide/ 199 8... | 


7J 


/prog info/SFA/StudentGuide/ 1998... | 


X| 


/prog_info/SF A/StudentGuide/ 1 998. .. | 


7] 


/ofTices/OPE/Studenis/student.html I 


4H 


/offices/OPE/express.html | 


3 ] 


/PDFDocs/StdGde98.pdf ] 


3^ 


Subtotal (/prog_info/SFA/StudentGuide«.) | 


is] 


/ofTices/OPE/Students/ 


/offices/OPE/Students/student.html | 


15 1 


/offices/OPE/Students/apply.html | 


5\ 


/offices/OPE/Professionals/ | 


4 



/offices/OPE/Students/fedaid.html 



/offices/OPE/Students/help.html 



/prog_info/S F A/FYE/ 



4 

X 

X 



/prog_info/SF A/StudentGuide/ 1 998-9/ 



/ 



/offices/OPE/Funding/ 



/offices/OPE/Studenls/ 



3 

"49" 



Subtotal (/offices/OPE/Students/) 



/ 



/offices/OPE/express.html 



3 

X 

X 

X 

X” 

IT 



/prog_info/SF A/StudentGuide/ 1 998-9/ 



/ 



/fmaid.html 



/prog_info/SF A/StudentGuide/ 1 998.. 



Subtotal (/) 



/oflices/OPE/Students/apply/fexprcss*.. 



/offices/OPE/Students/apply/dwnl . , 



/PDFDocs/98 fafsafmaiyform.pdf 



Subtotal (/officcs/OPE/Students/apply...) 
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/prog _info/SFA/StudentGuide/1998-9/e... 






/prog info/SFA/StudentGuide/1998... 


4 


/prog info/SFA/StudentGuide/ 1 998... 


2 


Subtotal (/prog^info/SFA/StudcntGuide...) 


6 


/offlces/OPE/Students/studenthtml 




/DirectLoan/ 


2 


/offices/OPE/Students/hopegd.html 


2 


/prog info/SFA/FYE/ 


2 


Subtotal (/ofTices/OPE/Students/stude...) 


6 


/pro&_info/SFA/StudentGuide/1998-9/f... 




/prog info/SFA/StudentGuide/ 1998... 


5 


Subtotal (/prog^info/SFA/StudentGulde...) 


5 


/prog^info/SFA/ 




/prog info/SFA/FYE/ 


2 


/prog_info/SFA/LSA/ 


2 


Subtotal (/prog^info/SFA/) 


4 


Subtotal (/prog^info/SFA/StudentGulde/) 


695 


/free/ 




/free/subjecthtml 




/frcc/s-scienc.html 


53 


/frcc/s-edtech.html 


49 


/frcc/s-social.html 


40 


/fi^s-math.html 


34 


/frcc/s-lanart.html 


31 


/free/s-arts.html 


24 




/free/s-voced.html 


20 




/free/s-forlan.html 


17 




/free/s-health.html 


15 




/free/subject.html 


13 




Subtotal (/free/subjecthtml) 


296 




/free/whathtml 






/free/sitemap. html 


68 




/free/subject.html 


68 




/free/kids.htxnl 


11 




/free/s-social.html 


9 




/free/ 


7 




/frcc/partner.html 


5 




/frcc/s-edtech.html 


5 




/frce/what.html 


5 




/pubs/parents/intemet/ 


3 




/frcc/s-all.html 


A 


\ 


Subtotal (/free/whathtml) 


18i 


r 
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/pubs/parents/internet/ 


1 


/pubs/parents/intemet/ 




/free/s*health.html 




/pubs/parents/intemet/message.html 


r| 


/pubs/parents/intemet/support.html 




/pubs/parents/intemet/title.html 




Subtotal (/pubs/parents/internet/) 


6 1 


/free/s-lanarthtml 


1 


/inits/americareads/ 


3 I 


/inits/americarcads/rcadwnow.htmi 


2 I 


Subtotal (/free/s-lanarthtml) 




Subtotal (/free/) 


613^ 


/ofriccs/OPE/cxpress.html 


1 


/offices/OPE/Students/apply/fexpress... 


1 


/offices/OPE/Students/appiy/dwnl... 


141 I 


/PDFDocs/98fafsafmal/form.pdf 




/ 


4 1 


/offices/OPE/express.html 


4 I 


/offices/OPE/Students/ 




/offices/OPE/Students/apply/fexp... 


4 1 


/prog_info/SFA/StudentGuide/ 


2 1 


/prog_info/SF A/StudentGuide/ 1 998-9/ 




/officcs/OPE/Professionals/ 


1 1 


Subtotal (/ofIices/OPE/Students/apply~.) 


168| 


/PDFDocs/98fafsarinal/form.pdf 




/PDFDocs/98fafsafinal/inst.pdf 


66 1 


/PDFDocs/98fafsafinal/form.pdf 


29^ 


/offices/OPE/Students/ 


13 1 


/offices/OPE/Students/apply/fexp... 




/offices/OPE/express.html 


5| 


/officcs/OPE/Students/apply.html 


2 1 


/officcs/OPE/Students/studenthtml 


2 1 


/ 


rj 


/BASISDB/TITLE4/search/sf 


r| 


/DirectLoan/consoiid.html 


1 1 


Subtotal (/PDFDocs/98fafsarmal/form.pdO 


129 1 


/offlces/OPE/Students/ 




/offices/OPE/Students/student.htinl 


28 1 


/offices/OPE/Students/apply.html 


26 1 


/offices/OP E/Students/ fedaid.html 


14^ 


/ofnces/OPE/Students/hopegd.htnnl 


11 1 


/BASISDB/TITLE4/search/sf 


s 1 
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/offices/OPE/Students/help.html 


5 1 


/offlces/OPE/Partners/ 


4I 


/offices/OPE/Students/ 




/PDFDocs/98 fafsafinal/form.pdf 




/ 


3 1 


Subtotal (/offlces/OPE/Students/) 


104 1 


/PDFDocs/98fafsafinal/instpdf 




/PDFDocs/98fafsafinal/instpdf 


14 1 


/PDFDocs/98fafsafinal/form.pdf 


9| 


/offices/OPE/Students/ 


5 1 


/offices/OPE/exprcss.html 


T] 


/fimding.html 


r| 


/offices/OPE/Students/hopegd.html 


1 I 


/offices/OPE/Students/student.html 


ij 


/prog info/SFA/StudentGuide/1998-9/ 


1 1 


/prog info/SFA/StudentGuide/1998... 


1 1 


SubtoUl (/PDFDocs/98fafsafinal/instpdO 


35^ 


/ofIices/OPE/express.htiiil 


1 


/PDFDocs/98fafsafinal/form.pdf 


8 1 


/offices/OPE/exprcss.html 


4 1 


/offices/OPE/Students/apply/fexp. . . 


3 I 


/offices/OPE/Students/apply/dwnl. . . 


2| 


/ 


‘1 


/DirectLoan/students.html 




/EdRes/EdFed/ 




/offices/OPE/Students/ 


* 1 


/PDFDocs/98fafsafinal/inst.pdf 


* 1 


SubtoUl (/offices/OPE/express.html) 


22\ 


/ 


1 


/ 


3| 


/offices/OPE/express.html 


3 1 


/PDFDocs/98 fa£safinal/form.pdf 


3 1 


/PDFDocs/98fafsafinal/instpdf 


2 1 


/BASISDB/TITLE4/search/sf 


r| 


/BASISDB)TITLEIV/search/SF 


T| 


/finaid.html 


Fj 


/offices/OPE/ 




/offices/OPE/Students/apply/fexp... 




/pros info/SFA/StudentGuide/ 


r| 


Subtotal (/) 


17^ 


/offices/OPE/Students/studenthtml 


1 


/prog_info/SFA/LSA/ 


5 1 
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/prog_info/SFA/FYE/ 


4 1 


/DirectLoan/ 


\ 


/offices/OPE/Students/fedaid.html 


1 1 


/offices/OPE/Students/ file4b.html 


n 


/offices/OPE/Students/student.html 




/prog_info/SF A/StudentGuide/ 


1 1 


Subtotal (/offices/OPE/Students/stude...) 


14 I 


/offices/OPE/Students/apply/dwnload9... 


1 


/offices/OPE/Students/ 


2 1 


/offices/OPE/Students/apply/fexp... 


2 I 


/ 


7] 


/databases/ERIC_Digests/ed4 1452 1 ... 


1 1 


/offices/OPE/express.htmi 


M 


/offices/OPE/Students/ fedaid.html 


1 1 


/offices/OPE/Students/help.html 


> 


/PDFDocs/98fafsafinal/form.pdf 


> 


/prog_info/ERIC/ 


r| 


/prog_info/SFA/FYE/stafford.html 


1 1 


Subtotal (/ofrices/OPE/Students/apply...) 


12 1 


/oflices/OPE/ 


1 


/offices/OPE/express.html 


2 1 


/offices/OPE/ 


> 


/offices/OPE/finaid.html 


n 


/offices/OPE/home.htmi 




/offices/OPE/Paitners/ 


> 


/offices/OPE/Professionals/ 


> 


/offices/OPE/Students/ 


> 


/offices/OPE/Students/apply/dwnl... 


> 


/offices/OPE/Students/other.htxnl 


> 


/prog_info/SFA/StudentGuide/ 


1 1 


Subtotal (/ofAces/OPE/) 


11 1 


/prog_info/SFA/StudentGuide/ 


1 


/prog_info/SFA/StudentGuide/ 1998-9/ 


3 1 


/prog_info/SFA/StudentGuide/ 1 998 .. . 


2 1 


/DirectLoan/ 




/offices/OPE/DCS/ 


1 


/offices/OPE/express.html 


1 


/offices/OPE/Students/ 




/offices/OPE/Students/student.html 


> 


Subtotal (/prog_info/SFA/StudentGulde/) 


1^1 


Subtotal (/ofnces/OPE/express.html) 


522 1 


Total (First requests) 


5331 1 



o 
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Appendix IV-3, HTML Search Form using GET 
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Appendix IV-3, HTML Search Form using GET 

<form action="http: //165 . 224 . 220 . 67 : 8765/results .html" method="GET"> 
<input type="hidden" name="col" value="site misc"><input type="hidden" 
name="qc" value="0"> 

< 1 — This is the table used only for the search form — > 

<table border="0" cellpadding="l" cellspacing="l" width="85%"> 

<tr> ' ■ 

<td>Fincl</tcl> 

<tdxselect name=*'qp" size='*l**> 

<option value="site : ed. gov url:-edoig url : -zsudiak'* selected>any 
do cument s < / opt i on> 

<option value=”url : legislation/ESEA 

url : legislation/G0ALS2000">legislative texts </option> 

<option value=”url : legislation/FedRegister">Federal Register documents 
</option> 

<option value="url : PressReleases">press releases </option> 

<option value=*'url : databases/Eric_Digests">ERIC Digests </ option> 
<option value='*url:nces.ed.gov">lSIational Center for Education 
Statistics documents </option> 

<option value=*'url : MailingLists/EDInfo">EDInfo mailing list messages 
</option> 

</select> </td> 

</tr> 

<tr> 

<td>That contain : </td> 

<tdxinput type="text" name="qt" size="44”> </td> 

</tr> 

<tr> 

<tdxbrx/td> 

<tdxinput type=" submit" value="Find"> 

<a href="http://165.224.220.67:8765/help/">(Help with search)</a> </td> 
</tr> 

</table> 

<! — End of search form related table — > 

</form> 




Appendix 76 



Appendix IV-4, U.S. Department of Education Search Results 

Using GET 
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USER URL IS; http://165.224.220.67:8765/results.html?col=site+misc&qc=0&qp=site%3Aed.gov+ 
urI%3A%7Eedoig+url%3A%7Ezsudiak&qt=Drag+Programs 

7030 results found, top 100 sorted by relevance, for Drug Programs. 

1-10 



OSERS - RSA Training Program: Rehabilitation Counseling 

The usual educational requirements for a position as a Rehabilitation Counselor is a Master s degree. To 
obtain a Master's degree in Rehabilitation 

Counseling,..* - 

http://www.ed.gov/offices/OSERS/RSA/PGMS/RT/carcouns.html - size 4.3K - U.S. Department ot 



Education 



70% 



Creating Safe and Drug-Free Schools: Table of Contents 

For the user's convenience, this document is also available in Portable Document Format (33 IK) from the 
National Criminal Justice Reference 
System (NCJRS)... 

htto7/www ed.gOv/offices/OESE/ACTGUID/index.html - size 2.7K - U.S. Department of Education 

59% 



Facts on Safe and Drug-Free Schools 

Nearly 3 million thefts and violent crimes occur on or near school campuses every year. This equates to 
almost 16,000 incidents per school day or 
one every... 

htto7/www ed.gOv/updates/fact-209.html - size 12.9K - U.S. Department of Educanon 

56% 



05/06/97: Safe and Drug-Free Schools and Communities National Programs [OESE] 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID;fr06my97-104] - 
DEPARTMENT OF EDUCATION [CFDA 

Nos.:84.184F,84.184G,... ttc 

http://www.ed.gOv/legislation/FedRegister/annoimcements/1997-2/050697d.html - size 7.6K. - U.b. 

Department of Education 

55% 



08/19/97: Notice extending application deadline for the Safe and Drug-Free Schools and ... 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:frl9au97-28] == - 
DEPARTMENT OF EDUCATION 

[CFDANo. 84.184F;84.184G;... .,oxr itc 

http://www.ed.gov/legislation/FedRegister/announcements/1997-3/081997b.html - size 2.8K - U.b. 

Department of Education 

55 % 
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Drug Prevention Programs in Higher Education 

Chapter 534 Drug Prevention Programs in Higher Education (CFDA No. 84.183) I. Program Profile 
Legislation: Title IV, Part D (Section 5131) of 
the Safe... 

httoV/www ed.gov/pubs/Biennial/534.html - size 11.9K - U.S. Department of Education 

54% 



Drug-Free Schools and Communities State and Local Programs 

Part B (sections 5121-5127) of the Drug-Free Schools and Communities Act (DFSCA) of 1986, as 
amended (20 U.S.C. 3191-3197) (expires 
September 30, 1999).... 

http-//www ed.gov/pubs/Biennial/1 15.html - size 7.2K - U.S. Department of Education 

54% 



07/22/98: Office of Elementary and Secondary Education— Safe and Drug-Free Schools and ... 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr22jy98-l 1 1] [[Page 
39449]] .. department OF 
EDUCATION [CDFA ... 

http://www.ed.gov/legislation/FedRegister/announcements/1998-3/072298c.html - size 7.3K - U.S. 
Department of Education 

54% 



05/06/97: Safe and Drug-Free Schools and Communities National Programs [OESE] 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCrD:ft06my97-103] - 
DEPARTMENT OF EDUCATION Safe 
and Drug-Free Schools and Commumties... 

http://www.ed.gOv/legislation/FedRegister/finmle/1997-2/050697c.html - size 21.5K - U.S. Department of 



Education 



54% 



ED34188631 Dec91 Are School-Based Dmg Prevention Programs Working? ERIC Digest. 

Author: Mohai, Caroline E. ERIC Clearinghouse on Counseling and Personnel Services, Arm Arbor, Mich. 
THIS DIGEST WAS CREATED BY 
ERIC, THE EDUCATIONAL RESOURCES... 

http7/www ed.gov/databases/ERIC_Digests/ed341886.html - size 1 1.7K - U.S. Department of Education 

54% 
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Appendix IV-5, HTML Search Form using POST 
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Appendix IV-5, HTML Search Form using POST 

<forni action="http://165.224.220.67:8765/results.htnil" method="POST"> 
<input type="hidden" nanie="col" value="site niisc"><input type="hidden" 
name="qc" value="0"> 

< 1 — This is the table used only for the search form > 

<table border="0" cellpadding="l" cellspacing="l" width="85%"> 

<tr> 

<td>Find</td> 

<tdxselect name="qp" size="l"> 

<option value="site : ed. gov url:~edoig url : ~zsudiak" selected>any 
documents</ option> 

<option value="url : legislation/ESEA 

url: legislation/G0ALS2000">legislative texts </option> 

<option value="url:legislation/FedRegister">Federal Register documents 
</option> 

<option value— **ur 1 : Pre s sReleases >pres s releases </option^ 

<option value="url:databases/Eric_Digests">ERIC Digests </option> 
<option value="url:nces .ed.gov">National Center for Education 
Statistics documents </option> 

<option value="url:MailingLists/EDInfo">EDInfo mailing list messages 
</option> 

</select> </td> 

</tr> 

<tr> 

<td>That contain : </td> 

<tdxinput type="text" name="qt" size="44"> </td> 

</tr> 

<tr> 

<tdxbr></td> 

<tdxinput type="submit" value="Find"> 

<a href="http://165.224.220.67:8765/help/">(Help with search)</a> </td> 
</tr> 

</table> 

<! — End of search form related table — > 

</form> 
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Appendix IV-6, U.S. Department of Education Search Results 

Using POST 
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Appendix IV-6, U.S. Department of Education Search Results Using POST 

USER URL IS: http;//165.224.220.67:8765/results.html 

7030 results found, top 100 sorted by relevance, for Drug Programs. 

MO 

OSERS - RSA Training Program: Rehabilitation Counseling 

The usual educational requirements for a position as a Rehabilitation Counselor is a Master's degree. To 
obtain a Master’s degree in Rehabilitation 
Counseling,... 

http://www.ed.gov/offices/OSERS/RSA/PGMS/RT/carcouns.html - size 4.3K - U.S. Department of 
Education 

70% 



Creating Safe and Drug-Free Schools: Table of Contents 

For the user's convenience, this document is also available in Portable Document Format (331K) from the 
National Criminal Justice Reference 
System (NCJRS)... 

http://www.ed.gov/offices/OESE/ACTGUlD/index.html - size 2.7K - U.S. Department of Education 

59% 



Facts on Safe and Drug-Free Schools 

Nearly 3 million thefts and violent crimes occur on or near school campuses every year. This equates to 
almost 16,000 incidents per school day or 
one every... 

http*// www.ed.gov/updates/fac t-209.html - size 12.9K - U.S. Department of Education 

56% 



05/06/97: Safe and Drug-Free Schools and Communities National Programs [OESE] 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr06my97-104] - 



DEPARTMENT OF EDUCATION [CFDA 
Nos.: 84.184F, 84.184G, ... 

http://www.ed.gOv/legislation/FedRegister/announcements/1997-2/050697d.html - size 7.6K - U.S. 
Department of Education 

55% 



08/19/97: Notice extending application deadline for the Safe and Drug-Free Schools and ... 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:frl9au97-28] = - 



DEPARTMENT OF EDUCATION 
[CFDA No. 84.184F; 84.184G; ... 

http://www.ed.gov/legislation/FedRegister/announcements/l997-3/081997b.html - size 2.8K - U.S. 
Department of Educatic n 

55% 



Drug Prevention Programs in Higher Education 
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Chapter 534 Drug Prevention Programs in Higher Education (CFDA No. 84.183) I. Program Profile 
Legislation: Title IV, Part D (Section 5 131) of 
the Safe... 

httn7/www ed gov/pubs/Biennial/534.html - size 11.9K - U.S. Department of Education 

54% 



Drug-Free Schools and Communities State and Local Programs 

Part B (sections 5121-5127) of the Drug-Free Schools and Commumties Act (DFSCA) of 1986, as 
amended (20 U.S.C. 3191-3197) (expires 
September 30, 1999).... 

htto7/www ed.gov/pubs/Biennial/1 15.html - size 7.2K - U.S. Department of Education 

54% 



07/22/98: Office of Elementary and Secondary Education-Safe and Drug-Free Schools and ... 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr22jy98-l 1 1] [[Page 
39449]] - department OF 

EDUCATION [CDFA ... 

http://www.ed.gov/legislation/FedRegister/aimouncements/1998-3/072298c.html - size 7.3K - U.S. 
Department of Education 

54% 



05/06/97: Safe and Drug-Free Schools and Communities National Programs [OESE] 

From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:ft06my97-103] - 
DEPARTMENT OF EDUCATION Safe 
and Drug-Free Schools and Conununities... 

http://www.ed.gOv/legislation/FedRegister/finmle/1997-2/050697c.html - size 21.5K - U.S. Department of 



Education 



54% 



ED341886 31 Dec 91 Are School-Based Dmg Prevention Programs Working? ERIC Digest. 

Author: Mohai, Caroline E. ERIC Clearinghouse on Counseling and Persoimel Services, Ann Arbor, Mich. 
THIS DIGEST WAS CREATED BY 
ERIC, THE EDUCATIONAL RESOURCES... 

htto7/www ed gov/databases/ERIC_Digests/ed34 1886.html - size 1 1.7K - U.S. Department of Education 

54% 




Appendix 84 



Appendix IV-7, OESE Pages and Images Not Found 
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Appendix IV-7, OESE Pages and Images Not Found 



Pages Not Found 

http://www.ncbe. gwu.edu/obemla/t 2 in/ - Is Referenced By: 
http://www.ed.gov/offices/OESE/asst.html 



http://www.neca.org/funds - Is Referenced By: 

http://www.ed.gov/offlces/OESE/hot.html 



http://www.ed.gov/ofrices/OESE/news_archive/contact.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/news_archive/l-97.html 



http://www.ed.gov/offices/legislation/FedRegister/announcements/1996-4/122796a.html - Is Referenced 

By: 

http://www.ed.gov/offices/OESE/news_archive/2-97.html 



http://www.ed.gov/offices/OESE/news_archive/ACTGUID/index.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/news_archive/4-97.html 



http://www.ncbe.gwu.edu/iasconferences/registration.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/events_archive/index.html 



http://www.ncbe.gwu.edu/iasconferences/regform.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/events_archive/index.html 



http://www.ed.gOv/legislation/GOALS2000yThe Act/intro.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/leg_reg.html 



http://www.ed.gov/offices/OESE/MEP/eventhtml - Is Referenced By: 
http://www.ed.gov/offices/OESE/MEP/events.html 



http://www.ed.gov/offices/OESE/CEP/happening.html - Is Referenced By: 
http://www.ed.gov/offices/OES^CEP/may22.html 
http://www.ed.gov/offices/OESE/CEP/may22.html 
http://www.ed.gov/offices/OESE/CEP/may 1 5.html 
http://www.ed.gov/offices/OESE/CEP/may 1 5 .html 



http://www.ed.gov/offices/OESE/MEP/PreliinGuide/legislation/ESEA/secl 1 1 l.html - Is Referenced By; 
http://www.ed.gOv/offices/OESE/MEP/PrelimGuide/ptla.html 



http://www.ed.govAegislation/ESEA/disadvan/times495.html" - Is Referenced By: 
http://www.ed.gov/offices/OESE/MEP/PrelimGuide/brochure.html 
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http://www.ed.gov/offices/OESE/MEP/PrelimGuide/www.ed.gov/offices/OESE - Is Referenced By: 
http://www.ed.gov/offices/OESE/MEP/PrelimGuide/brochure.html 



http://www.nwrel.org/national/regional-labs.html - Is Referenced By: 

http://www.ed.gov/offices/OESE/MEP/PrelimGuide/appendix.html 



http://www.enc.org/consortia.htinl - Is Referenced By: 

http://www.ed.gov/offices/OESE/MEP/PrelimGuide/appendix.html 



http://www.ed.gov/offices/OESE/StandardsAssessment/toc.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/StandardsAssessment/perfbased.html 
http://www.ecLgov/offices/OESE/StandardsAssessment/perftask.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/migrant.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/migrant.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/multiple.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/multiple.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/measure.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/measure.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/multms.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/sample.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/sample.html 
http://www.ed.gov/offices/OESE/StandardsAssessment/local.html 



http://www.ed.gov/offices/OESE/StandardsAssessment/ndex.html - Is Referenced By: 
http://www.ed.gov/offices/OESE/StandardsAssessment/approach.html 
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Images Not Found 

http://www.ed.gov/offices/OESE/news_archive/bkg.gif- Is Referenced By: 
http://www.ed.gov/offices/OESE/news_archive/l-98.html 
http://www.ed.gov/offices/OESE/news_archive/2-98.html 
http://www.ed.gOv/offices/OESE/news_archive/l -97 .html 
http://www.ed.gov/offices/OESE/news_archive/2-97.html 
http://www.ed.gov/offices/OESE/news_archive/3-97.html 
http://www.ed.gov/offices/OESE/news_archive/4-97.html 
http://www.ed.gov/offices/OESE/news_archive/7-97.html 



http://www.ed.gov/offices/OESE/events_archive/bkg.gif-Is Referenced By: 
http://www.ed.gov/offices/OESE/events_archive/index.html 



http://www.ed.gOv/offices/OESE/magnet/msap_apO.gif- Is Referenced By; 
http;//www.ed.gov/offices/OESE/magnet/cover.html 



http://www.ed.gov/offices/OESE/StandardsAssessment/guidan23.gif- Is Referenced By; 

http;//www.ed.gov/offices/OESE/StandardsAssessment/trans.html 



http://www.ed.gov/offices/OESE/StandardsAssessment/guidan26.gif- Is Referenced By: 
http://www.ed.gov/offices/OESE/StandardsAssessment/acct.html 



http://www.ed.gOv/offices/OESE/StandardsAssessment/guidan27.gif- Is Referenced By: 
http;//%vww.ed.gov/offices/OESE/StandardsAssessment/acct.html 



http;//www.ed.gov/offices/OESE/CEP/hands/happening_l.gif - Is Referenced By; 
http;//www.ed.gov/offices/OESE/CEP/may22.html 
http;//www.ed.gov/offices/OESE/CEP/may 1 5 .html 



http;//www.ed.gov/offices/OESE/StandardsAssessment/guidan25.gif - Is Referenced By: 
http://www.ed.gov/offices/OESE/StandardsAssessment/id.html 
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Appendix IV-8, Text of Suck.com Publication: 
http://www.suck.com/daily/98/09/22/daily.html 
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Appendix IV-8, Text ofSuck.com Publication: http://www,suckxom/daily/98/09/22/daily.html 

Suck. 

“a fish, a barrel, and a smoking gun” 
for 22 September 1998. Updated every WEEKDAY. 

Online: http://www.sucLcom/daily/98/09/22/daily.html 

Clown Act 

Lose the "the,” and it costs just short of a billion dollars a word: The Safe and Drug-Free Schools and 
Communities Act. Now 12 years old, it has allocated nearly US$6 billion of federal money to state and 
local governments for the purpose of - well, guess. Who could argue? Last time we checked, not too many 
anti-school-safety advocates were working the halls of Congress or making the rounds of the Sunday 
morning talk shows. Then, of course, there's the fact that schools really are a great deal safer today than 
they were 12 years ago, a distant era when the student body was less able to mount a decisive, armed 
defense. 

Just how safe are they? Ask Ralph Frammolino, the Los Angeles Times reporter who recently bothered to 
investigate how all that money was spent - a question with an awfully long answer. There's the "clown act 
promoting bicycle safety" in Arkansas, the $1,000 in bait and poles for field trips to a fishing hole in Utah, 
the $6,500 remote-controlled toy police car in Louisiana, the dunking booths in Pennsylvania, the new 
Pontiac Grand Prix for the always-stylish cops at the Los Angeles Unified School District. There's Miss 
Louisiana singing the theme song from Titanic to some unlucky children down in Jefferson Davis Parish. 

And then there's the magician who defended his anti-drug school assemblies with the helpful explanation, 
"We have a live duck in the show." 

The act really is an act. The departments of Justice and Education both released reports last year offering 
that conclusion, calling it "a relatively narrow range of intervention strategies, many of which have been 
shown either to not work ... or to have only small effects." The Congressional Budget Office suggested - 
unsuccessfully, of course - that maybe it was time to stop funding all those live duck and clown acts. And 
actual students handed in their own verdict on the effectiveness of the anti-drug portion of the act by 
getting stoned out of their gourds. As the Times explains, the number of eighth graders trying marijuana 
has more than doubled since the early 1990s. (In similarly consternating news, sales of Canibus CDs more 
than doubled just last week.) 

"Still," Frammolino reports, "US education officials insist the program is worthwhile and that schools are 
safe, with 90 percent of the nation's campuses never reporting any act of serious violence." Ah, yes, the 
Sun-Coming-Up Program accepts your thanks for this morning's sunrise. The program is worthwhile, and 
schools are safe. We're pissing off the balcony, and forest fires are down 10 percent in the Pacific 
Northwest Think of it as a kind of crossword puzzle or "Where's Waldo" game. 

But there's more to our favorite crisis-programming convolutions than their brainteaser value; the best ones 
both demonstrate and inspire tme courage. Remember A1 Gore on Letterman - the show, that is - all those 
years back, smashing the ashtray with a hammer? Sure you do, particularly since you've received 
occasional reminders that the administration was busily reinventing the federal government, making it more 
effective, faster, and less expensive. Except that a reporter at The Washington Monthly just checked into 
that new,’ more efficient federal government by trying to get a series of government agencies to perform 
simple tasks. Seth Grossman's story, in the September issue, details such complicated interactions as calling 
a cancer-information hot line, maintained by the Department of Health and Human Services ... and trying to 
get some information about cancer. Insert big surprise here: It didn't work. Grossman was - in calls to a 
dozen agencies - disconnected, promised tiformation by mail that never arrived, charged 1-900 fees of 35 
cents a minute for passport information, and instructed to call hot-line numbers that were no longer in 
service. His stretching-deep-for-the-lesson conclusion: "In far too many of these agencies, help lines seem 
to exist just for the sake of existing, not really because the agencies really want to provide anyone with any 
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help." But the reshaping of the federal administration has achieved one key goal: the reduction of the 
federal work force by 351,000. Symbols don’t need to be staffed - they just need a sign on the door. 

Writing in the 7 September issue of The New Republic, the less-than-dazzling Martin Peretz came up, 
surprisingly, with a bit of precise, perfect language - describing the critical political balance between 
"polemical outrage" and "functional indifference." He was writing about Clinton's policy toward Iraq, but 
he could just as easily have been writing about nearly any national politician - and just about any issue. 

Still, Clinton is very much the all-time champion at this game. The Monday after the president was 
distracted by personal issues, he appeared before the oh-so-seriously-named Council on Foreign Relations - 
or, rather, before a hastily assembled audience that was arranged in front of a sign with the council's name 
on it The president's staff had called the council on Wednesday of the previous week, explaining with 
some urgency that the president needed to give a major speech on economic policy right away. His cause: 
saving the world economy from a crash. Speaking somberly, wearing an expression of grim determination, 
the leader of the free world laid out his program for stopping a crisis that had already engulfed Russia and 
Asia and was emerging in Latin America. He had ordered the Secretary of the Treasury and the Chairman 
of the Federal Reserve to convene a meeting with their counterparts from other countries. And he didn't just 
want a meeting - he expected a report, too, "to recommend ways to adapt the international financial 
architecture to the 21st century." For substance, Clinton added a demand that Congress ship more money to 
the IMF, committing the United States to helping the collapsing economies pay their Visa bills with their 
MasterCards - or, we suppose, with American Express. Not much more advanced than casting shadows on 
the wall, but for a day the president was busy saving the world and didn't have time to dwell on less 
significant matters. 



We, on the other hand, would greatly prefer to dwell on the less significant matters, which tend to be 
simpler and much funnier over drinks - and which cost a whole lot less money, $40 million in 
investigations aside. Feel free to agree or disagree, but the duck-owning magicians among you should take 
at least one additional piece of information into consideration. Clinton is recommending that Congress 
increase funding for the Safe and Dmg-Free Schools and Communities Act in the next budget year - a 
position that really does take courage. 

Or something a lot like it. 

courtesy of Ambrose Beers 
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Appendix V-1, Customer Service Data Collection Forms 



Appendix 



Appendix V-1, Customer Service Data Collection Forms 

U.S. DEPARTMENT OF EDUCATION WEBSITE EVALUATION PROJECT 



Interview/Focus Group Participant Information Sheet 
Please answer the following questions: 

1 . Name 

Email Address 



2. Years working at the Department of Education (years) 

3. What Department ofEducation office do you work in? 

4. Job Title 

5. Primary Responsibilities (briej) 



6. In a general work day, I interact with the public answering questions (e.g. via phone or 
email) about information or services provided by the Department ofEducation for 

[ ] less than one hour 
[ ] 1-3 hours [ ] 6-8 hours 

[ ] 3-6 hours [ ] over 8 hours 



7. In a general work day, I spend the following amount of time on the following activities 

(please check one box for each activity): 

less than 1 hour 1-3 hours 3-6 hours 6-8 hours more than 8 hours 

Planning internet based services [ ] [ ] [ ] [1 

[ 1 

Implementing internet based services [] [] [] [1 

[ 1 

Maintaining internet based services [ ] [ ] [1 

[ 1 [1 
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8. For how many years have you developed internet based services for the Department 
of Education? (years) 



9. For how many years have you worked providing any kind of information services for 
the public with the Department of Education? (years) 



10. Are there any other individuals you think we should talk to? Please include names 
and phone extensions if possible. 



Thank you! 
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U.S. DEPARTMENT OF EDUCATION WEBSITE EVALUATION PROJECT 



User perspective interview guide 



USERS AND TASKS 

1. Through discussions with several administrators at the Department of Education, we 
identified several groups as users of the Department of Education website. These were; 
educational administrators, teachers and professors, researchers and analysts, students, 
parents, librarians, writers and reporters, policy makers and legislators. 

Does this categorization fit the customers you help? Are there other categories? Are 
there some categories of people that you don't help? 

2. We are also interested in identifying the types of tasks or questions that people have 
asked. Have you tabulated or categorized or analyzed the kinds of questions people ask 
you? If yes, could you 

describe that categorization? 

What are typical tasks that users engage in with the Department of Education website? 

(If they have tabulated question types, etc. try to get copy of that information-the less 
time spent on this question, the better, as it is more important to get on to how the 
intermediaries help them) 

INTERMEDIARY STRATEGIES 

INDIVIDUAL QUESTION (if group, skip to question 4B) 

3. Think back to the last question from a customer that you handled (through any point 
of contact— web, phone, etc.). Describe it. How did you go about answering it? (It 
doesn't matter how the person contacted the agency but make sure that you record that 
info) 

If necessary, probe for additional detail on sources used, strategies used, things that the 
customer didn't know but needed to, explanations provided by the intermediary, questions 
asked by the intermediary etc. of 
the customers. 

Individual 4 A. The purpose of asking you about a particular instance was to begin to 

get you think ing specifically about how you help customers. What types of resources, 
expertise, knowledge, understandings do you use as you work with customers? (Try to 
get agency-specific examples.) (Go to question 5.) 
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Group 4B. As part of this round of our investigations, we are gathering data on how 
you help customers. We will use this information to help us design on-line tools to help 
people access statistics when they don't interact with you or other intermediaries. Wh at 
types of resources, expertise, knowledge, understandings do you use as you work with 
customers? (Try to get agency-specific examples.) (Go to question 5.) 

5. What kinds of questions do you ask users in order to help yourself understand what 
they want? 

(Note Q. 5 will get at similar info to q. 4 but also will shed light on what the intermediary 
perceives as important dimensions of tasks/questions to understand) 
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U.S. DEPARTMENT OF EDUCATION WEBSITE EVALUATION PROJECT 

Intcrvicw/Focus Group Participant Infomation Sheet page 2 



CONCLUSION (questions to ask if time, though question 7, 8 should always be asked) 

6 In the first stage of the project, we found that users often had expectations about the 
agency and the data that influenced how they asked their questions, how they perceived 
the agency etc. What kinds of expectations do users have about your agency and data? 



7. What we have been trying to understand today are the ways in which you help 
users, the types of strategies you use to answer their questions, the resources employed, 
etc. We are looking for ways we can incorporate 

some of this expertise on websites, particularly the Department of Education website, so 
that the public who come to the agency via the web may take advantage of your 
knowledge, even though they may not interact with you "in real time." Is there anythmg 
else that we should know that would help us understand? 



8. Are there any additional people in your agency that you think we might wish to 
contact? Get contact information. 

9. Are there any other comments? 
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Appendix V-2, Task Based Evaluation Forms 
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Appendix V-2, Task Based Evaluation Forms 



EDEVAL TASK BASED USER SURVEY 
November 1998 

Interviewer instructions for conducting task based analysis. 

1 Set up web browser and load the homepage of the research site. Clear history to reset 
links. 

2. See A under participant instructions below. Thank the participants for coming, restate 
the purpose of the study and provide confidentiality statement, and ask if there are any 
questions. 

3. Read instructions aloud (see B below) 

4. Pass out WebMAC questionnaire 

5. Allow participants to begin browsing 

6. Alert participants after the passage of every ten minutes 

7. Stop the participants after 45 minutes. 

8. After completion of test, spend a few minutes with the participants answering 
participant questions. 

9. Ask the participants to complete the questionnaire (see C below). 

10. Thank the participant for their time and begin the focus group session (see D below). 
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Participant Instructions. 



Ask participants to excuse you for reading these instructions verbatim. While this may 
be a bit awkward, it will ensure that we do not mix up any of the instructions. 

A. Introduction to website evaluation 

Thank you for agreeing to. participate in this study. The purpose of this study is to 
evaluate the Department of Education websites. Your participation is extremely helpful 
as it allows us to observe how an “interested” user interacts with the Department of 
Education website. 

We are going to use a combination of techniques to collect data during this session. 

First, we will ask you to answer some questions using the website as an information 
source. Second, we have a questionnaire for you to complete. Finally, we will discuss 
your experiences all together in a group. 

We want you to understand that we are evaluating the website, not you, or your use of the 
website. In this regard, it is not really possible for you to make a mistake or do 
something wrong. To the extent possible, we hope that you conduct your visit just as you 
normally would if you were exploring on your own at home or work. 

Although we don’t anticipate confidentiality being an issue, be assured that all notes and 
survey forms will only be accessed by the survey team and will be destroyed after the 
study is completed. Study findings may be retained but will not include personal 
identifiers. 

Are you still willing to participate in this study? 

Do you have any questions so far? 

Now we would like you to fill out the demographic portion of the survey, then take some 
time to read the rest of the questionnaire so that you will understand the types of 
questions that we will be asking you to answer. 

<pause> 

Do you have any questions? 

B. Task Based Evaluation 

We have given yoii four tasks or questions to answer by using the Depa^ent of 
Education website. Answers to each question exist somewhere on the site. The purpose 
of the questions is to provide you some structure during your exploration of the site. Our 
interests center on your impressions of the site, not your success in answering the 
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c^ucstions. cncours§c you to explore 3iiy psrts of the site th&t you think would, be 
helpful in answering the questions. 

<instructions from scenario sheet) Use any features of the www.ed.gov website to 
answer the following questions. After you complete your searching, we will ask you to 
fill out a questionnaire and we will also ask you questions about your impressions of the 
website. 

As you search, please make not of the titles (not the urls) of the pages you move through 
as you find your answers in the table provided for each question. As an example, we will 
start you on the Department of Education home page, so we have printed Department of 
Education homepage as your first step in Table 1. Also, please note the title (not the url) 
of the page where you found the answer(s) directly below the table. Feel free to ask for 
help if you cannot determine the title of a page. 

Given that we have a limited amount of time however, we suggest that you only spend a 
maximum of 10 minutes working on each question. We have designed the questions to 
encourage you to explore different parts of the website. By working on all four 
questions, you will get the broadest possible view of the website and will be better able to 
formulate an opinion about overall usefulness of the site. Therefore, we will alert you 
after each ten minute period passes. You are not required to move on at this point 
however. If you complete all the questions prior to the 45 minutes allotted, please use 
any remaining time to further explore the site. 

We do ask that you primarily spend your time navigating the site to locate information 
and gain an understanding of the site structure rather then spending a great amount of 
time reading or downloading a particular document. Of course, looking over identified 
text to verify that you are interested in it or not is certainly appropriate as we do ask some 
questions concerning the quality of the website’s content. If you find something that you 
are personally interested in, please just bookmark the page and we can come back and 
retrieve it after the session is completed. 

Let’s start the session. We will cut you off at the end of 45 minutes in order to begin the 
focus group. 

To summarize: During the session we will prompt you every ten minutes to encourage 
you to work on all four questions. Also, we may urge you to continue exploring the site 
if you become bogged down in a particular section. If you have extra time at the end, we 
encourage you to continue exploring the site. You will answer the questionnaire in your 
possession at the end of the 45 minute period. 

Any final questions? 



Proceed. 



C. WebMAC questionnaire 
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Now that we have finished answering the questions and looking through the website, we 
would like you to fill out the following questionnaire regarding your experiences. The 
questionnaire has two pages. The questionnaire asks you to indicate to what degree you 
agree or disagree with a series of statements. Look carefully at the scale at the top of the 
questionnaire. A 3 indicates that you strongly agree. A 2 that you somewhat agree. A 1 
that you somewhat disagree. Finally a 0 indicates that you strongly disagree. If you don't 
feel you can answer the questions, please mark NA for not applicable. If you don't 
understand a question, please ask one of us for help. 

You may look at the website while you fill out the questionnaire. However, please finish 
the questionnaire within 1 0 minutes. 

Are there any questions? 

Please proceed. 

E. Focus Group 

The purpose of this focus group is to gather information from you regarding your 
experiences with the Department of Education website. 

Please take a moment to write down the top four weaknesses of the site, based on your 
explorations. 

<The researcher then wrote all of the weaknesses up on the chalkboard, noting repeats> 

To what extent do you agree with these weaknesses? 

What should be the top four? 

What should the purpose of the Department of Education homepage be? 

Do you wish to discuss anything else? 

Thank you very much for participating. The Department of Education will use your input 
to help redesign several aspects of their site. 
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Study Participant Demographics 



Name 

Age [ ] under 20 [ ] 20-30 [ ] 30-40 

[ ]40-50 [ ] 50-60 [ ] 60 and over 

Professional Experience 

Do you have a degree in education? Y N 

If yes, what is the degree(s)? BS MS EdD 

How many years experience do you have as an educator? 

How many years experience do you have as educational administrator? 

How many years experience do you have as a professional educational information 
specialist? 

Internet Use Experience 

Number of Years using the WWW 

Estimated Number of Web Pages Designed 

How would you rate your skill in navigating the WWW? 

Low High 

1 2 3 4 5 

How would you rate your experience using Department of Education Web pages? 

Low High 

1 2 3 4 5 

How would you rate your experience evaluating the design of Websites? 

Low High 

1 2 3 4 5 

How would you rate your experience evaluating the content of Websites? 

Low High 

1 2 3 4 5 
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Appendix V-3, Log Files for Task-Based Test 
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Appendix V-3, Log Files for Task-Based Test, Group 1 



syrul82-075 . syr . edu [ 05/Nov/ 1998 : 19 : 48 : 0 7 /Welcome/overview. html 

□ 

syrul82-075.syr.edu[05/Nov/1998:19:49:27/offices/OESE/SDFS/ 

□ 

yrul82-075. syr. edu [OS/Nov/1998: 19: 49: 56/of fices/OESE/SDFS/grants .html 

syrul82-075 . syr. edu [ 05/Nov/ 1998 : 19 : 54 : 13 /stats . html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 08 : 4 4 /programs . html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 09 : 10 /Welcome/overview . html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 11 : 08 /legislation/ESEA/ index. html 

syrul82-075 . syr. edu [ 05/Nov/ 1998 : 20 : 15 : 54/offices/OERI /At-Risk/ 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 23 : 4 1/news . html 

syrul82- 

075 . syr . edu[05/Nov/1998 :20 : 24 : 5 0/legislation/FedRegister /announcements/ 
1998-4 /110398a. html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 25 : 4 5/EdRes/index . html 
syrul82-075 . syr. edu [ 05/Nov/ 1998 : 20 : 49 : 01/Picks/picsbg . jpg 
syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 49 : 00 /Picks/index. html 
syrul82-07 5 . syr . edu [ 05/Nov/ 1998 : 19 : 47 : 12 /programs . html 
syrul82-075 . syr . edu [ 05/Nov/ 1998 : 19 : 53 : 4 8 / 

syrul82-075 . syr. edu [05/Nov/ 1998 : 19 : 53 : 52 /icons /doeseal . jpg 
syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 14 : 00/inits . html 
syrul82-07 5 . syr . edu [ 05/Nov/ 1998 : 20 : 16 : 52/funding . html 

syrul82-075 .syr .edu[05/Nov/1998 :20: 17 : 4 5/of fices/OBEMLA/ funding . html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 44 : 15/topicsaz . html 

syrul82-075 . syr . edu [ 05/Nov/ 1998 : 20 : 4 4 : 4 1/dirs . html 

syrul82-07 6 . syr . edu [ 05/Nov/ 1998 : 18 : 21 : 4 5 /icons /doeseal . jpg 

syr ul 82-07 6 . syr . edu [ 05/Nov/ 1998 : 18 : 2 1 : 4 2 / 

syrul82-076. syr. edu [ 05/Nov/ 1998 : 19 : 47 : 04 /programs . html 

syrul82-076. syr. edu [05/NOV/1998: 19:50: 4 O/pubs/EDInitiatives/98/ 98-10- 

29.html 

syrul82-07 6 . syr . edu [ 05/Nov/ 1998 : 19 : 52 : 14 /Pres sReleases/ 10- 
1998/safety, html 

syrul82-076.syr.edu [05/Nov/ 1998 : 19: 55 : 29/funding. html 
syrul82-07 6 . syr . edu [ 05/Nov/ 1998 : 19 : 57 : 13 /search . html 
syrul82- 

076 . syr . edu [05/Nov/1998 : 19 : 57 : 44/of fices/OSERS/IDEA/whatsnew. html 
syrul82- 

076.syr.edu[05/Nov/1998:19:58:40/offices/OSERS/IDEA/geninfo.html 

syrul82- 

076. syr.edu[05/Nov/1998: 19: 58: 58/of fices/OSERS/IDEA/overview. html 
syrul82- 

076. syr. edu [05/NOV/1998: 19: 59: 4 4/of fices/OSERS/IDEA/ updates. html 
syrul82- 

076. syr .edu [05/NOV/1998: 20:00 : 58/legislation/FedRegister/f inrule/1996- 

2/050996b.html 

syrul82- 

076. syr. edu [05 /Nov/ 1998 : 20 : 01 : 31/legislation/FedRegister/f inrule/ 
syrul82- 

076.syr.edu[05/Nov/1998:20: 01:28/legislation/FedRegister/finrule 
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Appendix V-3, Log Files for Task-Based Test 



syrul82- 

076. syr. edu [05/NOV/1998 : 20:02 : 26/legislation/FedRegister/f inrule/1998- 

4 /100198a. html 

syrul82-076. syr . edu [OS/Nov/1998 : 20 : 06: 39/legislation/ESEA/sec7144 . html 
syrul82-076. syr . edu [05 /Nov/1998 : 20: 07 : 18/legislation/ESEA 
syrul82-076 . syr . edu [05/Nov/1998 : 20 : 13 : 21/Welcome /overview. html 
syrul82-07 6.syr.edu[05/Nov/1998: 20: 14 : 33/legislation/ESEA/ index.html 
syrul82-076 . syr . edu [05/NOV/1998 : 20 : 24 : 12/off ices /OBEMLA/ funding. html 
syrul82~07 6 . syr . edu [ 05 /Nov/ 1998 :20: 24 : 39/offices/ OBEMLA/prop227 . pdf 
syrul82-076 . syr . edu [05 /Nov/ 1998 : 20 : 25 : 36/of f ices/OBEMLA/comgrant . html 

syrul82-076. syr. edu [05/NOV/1998 : 20: 43: 59/of fices/OBEMLA/aboutus .html 

syrul82-076 . syr . edu [05 /Nov/ 1998 :20:49:43/y2k/ 
syrul82-076. syr. edu [05/Nov/1998 : 19: 48 : 39/of f ices /OESE/SDFS / 
syrul82-076 . syr . edu [05 /Nov/1998 : 19 : 48 : 52/off ices /OESE/SDFS /grants . html 
syrul82~07 6 . syr . edu [ 05 /Nov/ 1998 : 20 : 07 : 19/legislation/ESEA/ 
syrul82-07 6. syr. edu [05 /Nov/ 1998: 20: 08: 05 /legislation/ 
syrul82-076. syr.edu[05/Nov/1998 :20: 12: 21/stats .html 

syrul82-076.syr.edu[05/Nov/1998:20: 44:04/offices/OBEMLA/pompasml. jpg 

syrul82-07 6 . syr . edu [05/Nov/1998 : 20 : 4 9 : 47/ of f ices/OCIO/ year / 
syrul82- 

076 . syr . edu [05 /Nov/1998 : 21 : 00 : 14 /databases /ERIC_Digests/ed4 03 101 . html 
syrul82-080 . syr . edu [ 05 /Nov/ 1998 : 18 : 18 : 57/ 

syrul82-080 . syr . edu [05 /Nov/ 1998 : 18 : 19: 01 /icons /doeseal . jpg 

syrul82-080.syr.edu[05/Nov/1998: 19: 47:32/pubs/EDInitiatives/98/98-10- 

29.html 

syrul82-080 . syr . edu [05 /Nov/ 1998 : 19 : 50:30/PressReleases/10- 
1998 /safety. html 

syrul82-080 . syr. edu [05 /Nov/1998 : 19 : 50: 58 /offices /OESE/SDFS /news .html 
syrul82-080 . syr . edu [ 05 /Nov/ 1998 : 19 : 53 : 58 /pubs /EDInit iatives /98 /98-10~ 
09 . html 

syrul82-080. syr. edu [05 /Nov/ 1998: 19: 55:02 /inits /construct ion/ 

syrul82-080 . syr . edu [05 /Nov/ 1998 : 19: 56: 12/DirectLoan/ 

syrul82-080 . syr . edu [05 /Nov/ 1998 : 19: 56: 21/prog_info/SFA/StudentGuide/ 
syrul82-080 . syr . edu[05/Nov/1998 : 20 : 01 : 58 /funding. html 
syrul82-080 . syr . edu [05 /Nov/1998 : 20 : 07 : 18 /search. html 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998:20:09:16/ of f ices /OSERS /IDEA/ geninf o . html 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998 : 20 : 09 : 57 /of f ices /OSERS /IDEA/q_and_a . html 
syrul82“080 . syr . edu [05 /Nov /1998 :20: 11: 13/offices/ OSERS / IDEA/ index . html 
syrul 82-08 0 . syr . edu [05 /Nov/ 1998 :20 : 11: 4 9 /programs .html 
syrul82-080 . syr . edu [05 /Nov/ 1998 : 20 : 12 : 16/Welcome/overview. html 
syrul82-080 . syr . edu [05 /Nov/1998 : 20 : 13 : 21/of fices/OESE/SDFS/ 
syrul82- 

080. syr. edu [05 /Nov/ 1998 : 20: 13: 57 /of fices/OESE/SDFS /aboutsdf. html 

syrul82“ 0 80 . s yr . edu [05 /Nov /1998:20: 14:07/ legis lat ion/ESEA/ sec4 011. html 

syrul82-080. syr. edu [05 /Nov/1998: 20: 15: 4 5/off ices/OESE/SDFS/grants. html 

syrul82-080. syr. edu [05 /Nov/1998 : 20: 21: 53/databases/gils/ 
syrul82-080 . syr . edu [05 /Nov/ 1998 : 20: 24 : 56/topicsaz.html 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998:19:49:18/ offices / OESE/SDFS / initiat i . html 
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syrul82-080. syr. edu[ 05 /Nov/1998: 19: 56: 13/DirectLoan/images/bgl. jpg 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998 : 20 : 08 : 05/MailingLists /EDInf o/msg00006 . html 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998 : 20 : 08 : 36/off ices /OSERS/ IDEA/ what s new . html 
syrul82- 

080 . syr . edu [ 05 /Nov/ 1998 : 20 : 10 : 31/of f ices /OSERS /IDEA/updates . html 
syrul82- 

080. syr . edu [05 /Nov/ 1998 : 20 : 10 : 43/of f ices /OSERS/IDEA/frnot ice . html 
syrul82-080 . syr . edu [ 05 /Nov/ 1998 : 20 : 16 : 09 /EdRes/ index. html 
syrul82-080 . syr.edu [05 /Nov/ 1998 : 20 : 16 : 39 /about . html 
syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 18 : 19 : 27 /icons /does eal . jpg 
syrul82-082 .syr.edu[05/Nov/1998 : 19: 06: 50/ 
syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 19 : 4 6 : 4 6/ funding . html 

syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 19 : 47 : 48 /of f ices /OESE/SDFS/grants . html 
syrul82- 

082 . syr . edu [ 05 /Nov/ 1998 : 19 : 53 : 24 /of f ices/OESE/SDFS/98abstra . html 
syrul82-082 . syr . edu [ 05 /Nov/ 1998 :19:54:09/offices /OESE/SDFS/grants . html 
syrul82- 

082. syr. edu [05 /Nov/ 1998 : 19: 54 : 4 9/offices /OESE/SDFS/about sdf.html 

syrul82-082 .syr. edu [05 /Nov/ 1998 : 19 : 57 : 42/ 

syrul82-082 . syr . edu [05 /Nov/ 1998 :19:58:21/stats. html 

syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 20 : 10 : 55/ 

syrul82-082 . syr . edu [05 /Nov/ 1998 : 20 : 12 : 14 /programs . html 

syrul82-082 . syr.edu [ 05 /Nov/ 1998 : 20 : 12 : 54 /Welcome /overview. html 

syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 20 : 15 : 05/off ices /OSERS /IDEA/ 

syrul82- 

082 . syr.edu [ 05 /Nov/ 1998 : 20 : 16 : 09/of f ices /OSERS/ IDEA/geninfo . html 

syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 20 : 17 : 30/ 

syrul82-082 . syr. edu [05 /Nov/1998 : 20 : 18 : 14/programs . html 

syrul82- 

082 . syr . edu [ 05 /Nov/ 1998 : 20 : 22 : 0 9 /legislat ion/ FedRegis ter /announcements/ 
1997-4 /I 11 4 97a. html 

syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 20 : 24 : 29/of f ices /OBEMLA/aboutus . html 

syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 20 : 27 : 28/ 

syrul82-082 . syr.edu [ 05 /Nov/ 1998 : 20 : 43 : 39/topicsaz . html 

syrul82-082 . syr.edu [ 05 /Nov/ 1998 : 20 : 43 : 4 9/topicsaz/topicsip . html 

syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 20 : 43 : 49/topicsaz/blue .jpg 

syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 20 : 4 4 : 14 /topicsaz/ index . html 

syrul82-082 .syr. edu [ 05 /Nov/ 1998 : 20 : 46 : 23 /topicsaz /index. html 

syrul82-082 . syr . edu [ 05 /Nov/ 19 98 : 18 : 19 : 25/ 

syrul82- 

082 . syr . edu [ 05 /Nov/ 1998:19:51: 27 /of f ices /OESE/SDFS / 98abstra . html 
syrul82-082 .syr. edu [ 05 /Nov/ 19 98 : 19 : 56 : 09/legislation/ESEA/sec4011 . html 
syrul82- 

082 . syr . edu [ 05 /Nov/ 1998 : 2 0 : 17 : 00 /of f ices / OSERS/IDEA/the_law . html 
syrul82-082 .syr. edu [05 /Nov/ 1998 : 20 : 19 : 38 /pubs /Biennial/ 2 02 . html 
syrul82-082 . syr . edu [ 05 /Nov/ 1998 : 20 : 24 : 29/of f ices /OBEMLA/pompasml . jpg 
syrul82-082 . syr.edu [ 05 /Nov/ 1998 : 20 : 43 : 50 /topicsaz /seal 95 . jpg 
syrul82-082 . syr . edu [ 05 /Nov/ 19 98 : 20 : 4 4 : 30/ topicsaz/ topiosip . html 
syrul82-084 .syr. edu [05 /Nov/ 1998 : 18 : 20 : 02/ 

syrul82-084 . syr . edu [05 /Nov/ 1998 : 18 : 20 : 04 /icons /doeseal .jpg 
syrul82-084 . syr.edu [ 05 /Nov/ 1998 : 19 : 46 : 38/f unding. html 
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syrul82-084 .syr.edu [05/Nov/ 1998: 19: 46: 38 /funding. html 

syrul82-08 4 . syr . edu [ 05 /Nov/ 1998 : 1 9 : 4 6 : 4 1 / funding . html 

syrul82-084 . syr . edu [05 /Nov/ 1998 : 19 : 48 : 01 /of f ices /OESE/SDFS/grants . html 

syrul82-084 .syr. edu [05/Nov/ 1998: 19: 50: 07 /off ices/OESE/SDFS/index. html 

syrul82- 

084 . syr . edu [05/Nov/ 1998 : 19 : 50 : 25 /offices /OESE/SDFS/aboutsdf . html 
syrul82-08 4 . syr . edu [ 05 /Nov/ 1998:20:00:34/ stats . html 
syrul82-084 . syr . edu [05 /Nov/ 1998 : 20 : 00 : 39/ stats . html 

syrul82-084.syr.edu [05 /Nov/ 1998: 20: 00: 4 9/stats. html 
syrul82-084.syr.edu [05 /Nov/ 1998: 20: 00: 50 /stats. html 

syrul82-084 . syr . edu [05 /Nov/ 1998 : 20 : 02 : 30/ stats . html 
syrul82-086 . syr . edu [05/Nov/ 1998:18:20:41/ 

syrul82-086.syr.edu[05/Nov/1998:18:20:43/icons/doeseal. jpg 

syrul82-086 . syr . edu [05/Nov/ 1998:20:02:32/ stats . html 
syrul82 — 08 6 . syr . edu [ 05 /Nov/ 1998:20:03: 11 /EdRes / index . html 
syrul82-086 . syr . edu [05/Nov/ 1998:20:03:37 /Beds 
syrul82-086 . syr . edu [05/Nov/ 1998:20:14:31/ 
syrul82-086 . syr . edu [05/Nov/ 1998:20:15:25/ idea 

syrul82-08 6. syr. edu [05/Nov/ 1998: 20: 15: 47 /programs. html 

syrul82-086 . syr. edu [05 /Nov/ 1998:20:24:57/ 

syrul82-086.syr.edu[05/Nov/1998:20:43:30/topicsaz.html 

syrul82-086.syr.edu[05/Nov/1998:20:44:32/topicsaz.html 

syrul82- 

086. syr . edu [05 /Nov/ 1998 : 20 : 45 : 52 /offices /OERI/BlueRibbonSchools/updates 

. html 
syrul82- 

086.syr.edu[05/Nov/1998:20:45:52/offices/0ERI/BlueRibbonSchools/banners 

/bnpr_upd. jpg 

syrul82- . /o • 

086. syr . edu [05 /Nov/ 1998 : 20 : 48 : 02/off ices /OERI /BlueRibbonSchools/ Scoring 

98-99.html 

syrul82- 

086.syr.edu[05/Nov/1998:20:49:50/offices/0ERI/BlueRibbonSchools/banners 

/bnpr_info. jpg 
syrul82- 

086.syr.edu[05/Nov/1998:20:49:50/offices/0ERI/BlueRibbonSchools/about.h 

tml 

syrul82- 

086. syr . edu [05/Nov/ 1998 : 20 : 52 : 18 /offices /OERI/BlueRibbonSchools/updates 

. html 

syrul82- 

086 . syr . edu [OS/Nov/1998 : 20 : 52 : 31 /of fices/OERI/BlueRibbonSchools/Scoring 

98-99.html 

syrul82- 

08 6. syr. edu [05 /Nov/ 1998: 20: 57: 4 9/off ices /OERI /BlueRibbonSchools /mainpg. 

html 

syrul82- . 

086. syr . edu [05/NOV/1998 : 20 : 57 : 57/of fices/OERI/BlueRibbonSchools/time . ht 

ml 

syrvl82- 

086.syr.edu[05/Nov/1998:20: 59: 12/offices/OERI/BlueRibbonSchools/prep.ht 

ml 
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syrul82- • u*. 

O86.syr.edu[05/Nov/1998 :20:59:22/offices/OERI/BlueRibbonSchools/time.ht 

ml 

syrul82-08 6. syr.edu [05/Nov/ 1998 : 20: 11: 50 /programs. html 

syrul82-08 6 . syr . edu [05/Nov/ 1998 : 20 : 25 : 27 / databases /ERIC_Digests / index/ 
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syrul82-07 5 . syr . edu [ 10 /Nov/ 1998 : 14 : 33 : 21/ 
syrul82-075 . syr . edu [ lO/Nov/1998 : 14 : 47 : 59/funding. html 

syrul82-075 . syr . edu [10 /Nov/ 1998 : 14 : 48 : 47/of f ices/OESE/SDFS/grants . html 
syrul82- 

075 . syr . edu [ 10 /Nov/ 1998 : 14 : 49 : 59/of fices/OESE/SDFS/ 98abstra . html 

syrul82-075 .syr.edu[10/Nov/1998: 14 :53: 13/offices/OESE/SDFS/ 

syrul82-075 . syr . edu [ 10 /Nov/ 1998 :14:55;13/ stats. html 

syrul82-075 .syr. edu [10 /Nov/ 1998: 15:01 :03/PressReleases/ 05- 

1998/gfsint .html , . .u i 

syrul82-075. syr. edu [ lO/Nov/1998 : 15: 02: 23/of fices/OSERS/RSA/rsakits. html 

syrul82-07 5 . syr . edu [ 10 /Nov/ 1998:15:04: 43 /pubs /AchGoal5 /osers . html 
syrul82- 

075 . syr.edu [ 10 /Nov/ 1998 : 15 : 06 : 06/of f ices /OSERS /IDEA/ what snew. html 
syrul82-075 . syr . edu [ 10 /Nov /I 998 : 15 : 06 : 30/off ices /OSERS /IDEA/ 
syrul82- 

O75.syr.edu[10/Nov/1998: 15: 07: 08 /of fices/OSERS/IDEA/f rnotice.html 

syrul82- . ^ i 

075 . syr . edu [10 /Nov/ 1998 : 15 : 07 : 19/off ices /OSERS /IDEA/ geninfo . html 

syrul82- . ^ i 

075 . syr . edu [10 /Nov/ 1998 : 15 : 07 : 39/off ices /OSERS /IDEA/ geninfo.html 

syrul82- 

075 .syr.edu [10 /Nov/ 1998 : 15 : 07 : 48/off ices /OSERS /IDEA/ overview.html 
syrul82-07 5. syr. edu [lO/Nov/1998: 15: 09: 39/programs. html 

syrul82-07 5 . syr . edu [ 10 /Nov/ 1998:15:16:18/ offices / OBEMLA/ about us . html 
syrul82- ^ 

075 . syr.edu [ 10 /Nov/ 1998 : 15 : 19 : 24 /legislation/GOALS2000 /TheAct/ secl012 . h 
tml 

syrul82- 

075 . syr.edu [ 10 /Nov/ 1998 : 15 : 20 : 50 /legislation/FedRegister /announcements/ 
1998 -2 /060398a .html 

syrul 82 - 075 . syr.edu[10/Nov/1998:15:21:05/offices/OSERS/RSA/ 
syrul82-07 5 . syr . edu [ 10 /Nov/ 1998 : 15 : 21 : 06/of f ices /OSERS /RSA/ 
syrul82-075. syr. edu [10 /Nov/1998 : 15:24: 55/about. html 
syrul82-07 5 .syr. edu [10 /Nov/ 1998: 15: 25: 21 /people. html 
syrul82-075 . syr . edu [ 10 /Nov/ 1998 : 15 : 25 : 44 / of f ices /OSERS/ 
syrul82-07 5.syr.edu [10 /Nov/ 1998; 15: 27: 18 /Awards/ 

syrul82-075.syr.edu[10/Nov/1998:15:27:43/pubs/EDInitiatives/98/98-10- 

2 9 . html 

syrul82-07 5 . syr . edu [ 10 /Nov/ 1998 : 15 : 32 : 34 / faqs . html 

syrul82-07 5 .syr. edu [10 /Nov/ 1998: 15:52 : 19/topicsaz. html 

syrul82-076 . syr . edu [lO/Nov/1998 : 14 : 32: 21/ 
syrul82-080 . syr.edu [ 10 /Nov/ 1998 : 14 : 48: 04/ 
syrul82-080 .syr.edu [ lO/Nov/1998 : 14 : 48 : 48/programs . html 
syrul82-080 . syr . edu [10/Nov/1998:14:54:57/offices/ OESE/SDFS / 
syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 14 : 56 : 4 6/ stats . html 
syrul82-080 . syr.edu [ 10 /Nov/ 1998 : 15 : 12 : 32/off ices /OSERS/ 

syrul82-080 . syr .edu [lO/Nov/1998 : 15 : 13 : 13/offices/OSERS/OSEP/index.html 
syrul82- 

080.syr.edu[10/Nov/1998: 15: 13:57/offices/OSERS/OSEP/Programs/PFA_Home.h 

tml 

syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 15 : 15 : 14 /faqs . html 

syrul82-080.syr.edu[10/Nov/1998:15:15:29/offices/OBEMLA/q_a.html 
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syrul82-080 . syr . edu[10/Nov/1998 : 15 : 16 : 00/of f ices/OBEMLA/q_al . html 
syrul82-080 . syr . edu [ lO/Nov/1998 : 15 : 17 : 39/Picks /index . html 
syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 15 : 18 : 25/topicsaz . html 
syrul82-080 . syr . edu [ lO/Nov/1998 : 15 : 18 : 59/topicsaz/index . html 
syrul8 2-080 . syr . edu [10 /Nov/ 19 98 : 15 : 19 : 4 1 /offices /OBEMLA/ 
syrul82-080 . syr.edu [ 10/Nov/ 1998 : 15 : 21 : 41/pubs/studies2 . html 
syrul82-080 . syr.edu [ lO/Nov/1998 : 15 : 22 : 12 /pubs /studies . html 
syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 15:29:21/ funding . html 

syrul82-080.syr.edu[10/Nov/1998:15:29:49/pubs/EDInitiatives/98/98-10- 

2 9 . html 

syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 15 : 30 : 17 / news . html 

syrul82-080 . syr . edu [10/Nov/ 1998 : 15 : 31 : 02/topicsaz/topicsip. html 

syrul82— 080 . syr . edu [ 10 /Nov/ 1998: 15:31: 10/offices/ OESE/SIP /magnet . html 

syrul82-080 . syr . edu [10/Nov/ 1998 : 15 : 31 : 14 /of f ices/OESE/magnet/sec-f . html 

syrul82-080 . syr . edu [ 10 /Nov/ 1998 : 15 : 32 : 22 / of f ices /OESE /magnet /index . html 

syrul82-080 . syr . edu[10/Nov/1998 : 15 : 32 : 31/of f ices/OESE/magnet/sec-b . html 

syrul82-082 . syr . edu [10 /Nov/ 1998 : 14 : 47 : 58/ 

syrul82-082 . syr . edu [10 /Nov/ 1998 : 14 : 48 : 41/programs . html 

syrul82-082 . syr . edu [10 /Nov/ 1998 : 14 : 49: 46/of fices/OESE/SDFS/ 

syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 14 : 50: ll/offices/ OESE /SDFS/ grants . html 

syrul82-082 . syr .edu[10/Nov/1998 : 14 : 51 : 10/of fices/OESE/SDFS/grants- 

archive/12-97 .html 

syrul82- 

082 • syr . edu [10 /Nov/ 1998 • 14 : 52 : 07 / offices/ OESE/ SDFS/ about sdf . html 
syrnl82-082 . syr . edu [ 10 /Nov/ 1998 : 14 : 53 : 14 /of fices/OESE/SDFS /index . html 
syrul82-082 . syr.edu [ lO/Nov/1998 : 14 : 54 : 26 /of fices/OESE/SDFS/grants . html 
syrul82-082 . syr . edu [10 /Nov/ 1998 : 14 : 54 : 53 /of fices/OESE/SDFS /news . html 
syrul82- 

082 . syr . edu [ 10 /Nov/ 1998 : 14 : 55 : 30/off ices /OESE/SDFS/research. html 
syrul82-082 . syr . edu [ 10 /Nov/1998 : 14 : 55 : 55/pubs /edpubs . html 
syrul82- 

O82.syr.edu[10/Nov/1998 :14 :56:22/offices/OESE/SDFS/programs.html 

syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 14 : 56 : 34 /of f ices /OESE /SDFS /links . html 
syrul8 2-08 2. syr. edu [10 /Nov/ 1998 : 14 : 57 : 03 /of f ices /OESE/SDFS /index . html 
syrul82- 

082 . syr .edu [10 /Nov/ 1998 : 14 : 57 : 07 /of f ices /OESE/SDFS /aboutsdf . html 

syrul82-082 .syr .edu[10/Nov/1998 : 14 : 57 : 48/off ices /OESE/ 

syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 14 : 58 : 53/ 

syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 05 : 03/ 

syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 07 : 05 / 

syrul82- 

082 . syr . edu [ lO/Nov/1998 : 15 : 08 : 4 5 /of f ices/OSERS/IDEA/speech_2 . html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 09 : 01/of f ices /OSERS/IDEA/ index . html 
syrul82- 

082 . syr . edu [ 10 /Nov/ 1998i 15:09: 44/ offices / OSERS/IDEA/ geninf o . html 
syrul82- 

082 . syr . edu [ 10/Nov/ 1998 : 15 : 10 : 00/of fices/OSERS/IDEA/ q_and_a . html 
syrul82- 

082 . syr. edu [ 10 /Nov/ 1998 : 15 : 11 : 13/ of f ices /OSERS/ IDEA/ what snew. html 
syrul82- 

082. syr. edu [ lO/Nov/1998 : 15: 12: 03/offices/OSERS/IDEA/f rnotice.html 

syrul82- 

082 . syr.edu [ 10 /Nov/ 1998 : 15 : 12 : 34 /of f ices /OSERS /IDEA/ the_law.html 
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syrul 82 - 

082 . syr . edu [ lO/Nov/1998 : 15:14: 09 /of f ices/OSERS/OSEP/Programs /TAD_IDEA . 
tml 

syrul 82 - 

082 . syr . edu [lO/Nov /1998 : 15 : 14 : 29 /of fices/OSERS/OSEP/Programs/PFA_Home . 



h 

h 



syrul82-082 . syr . edu [ 10 /Nov /1 998 : 15 : 15 : 43/ of f ices /OSERS/ IDEA/ index . html 
syrul82-082 . syr . edu [ 10/Nov/ 1998 : 15 : 16 : 06/ 

syrul82-082 . syr . edu [ lO/Nov/1998 : 15 : 17 : 09/of fices/OBEMLA/aboutus . html 
syrul82-082. syr. edu [10/Nov/ 1998: 15: 19: 12/PressReleases /05- 

1998 /gf sint . html 
syrul82- 

082 . syr . edu [10/Nov/ 1998 : 15 : 20 : 27 /of f ices/OESE/ACTGUID/gunf ree . html 
syrul82-082 . syr . edu [10 /Nov/ 1998 : 15 : 20 : 57/ 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 21 : 04 /programs . html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 21 : 24 / of f ices /OESE/SDFS / 
syrul82- 

082. syr. edu [ lO/Nov/1998 : 15: 21: 27 /off ices/OESE/SDFS/about sdf.html 
syrul82-082.syr.edu[10/Nov/1998: 15:21: 56/of fices/OESE/SDFS/index. html 
syrul82-082. syr. edu [ lO/Nov/1998 : 15:22: 15/offices/OESE/SDFS/links. html 

syru 18 2-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 22 : 39/ of f ices/OESE/ SDFS /news . html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 22 : 42 / of f ices /OESE/SDFS /me s sage . html 
syrul82-082 . syr . edu [10/Nov/ 1998 : 15 : 23 : 27 /programs . html 
syrul82-082,. syr . edu [10 /Nov/ 1998 : 15 : 24 : 48/ of f ices /OESE/SDFS/ 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 24 : 57 / of f ices/OESE/ 

syrul82-082 . syr . edu [ 10/Nov/ 1998 : 15 : 25 : 25/off ices /OESE/SDFS /grants . html 
syrul82-082 . syr . edu [10/Nov/ 1998 : 15 : 25 : 43 /of f ices/OESE/SDFS/index . html 
syrul82-082 . syr . edu [ lO/Nov/1998 : 15 : 27 : 59/Speeches/09-1994 /hispsum. html 
syrul82-082 . syr . edu [10/Nov/ 1998 : 15 : 28 : 23/ 

syrul82-082 . syr . edu [10/Nov/ 1998 : 15 : 32 : 47 /pubs /parents . html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 34 : 08 /pubs /Museum/ 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 34 : 17 / index . html 
syrul82-082 . syr . edu[10/Nov/1998 : 15 : 34 : 23 /programs .html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 34 : 40/off ices /OESE/SDFS/ 
syrul82- 

082. syr. edu [lO/Nov/1998: 15:34 : 44 /of f ices/OESE/SDFS/research . html 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 : 15 : 35 : 36/ 
syrul82-082 . syr . edu [ 10 /Nov/ 1998 ; 15 : 44 : 05/ 
syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 14 : 47 : 54/ 

syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 14 : 50 : 02/off ices/OESE/ SDFS/ 

syrul82-084.syr.edu[10/Nov/1998:14 : 50: 14 /off ices /OESE/SDFS /grants. html 

syrul82- 

084.syr.edu[10/Nov/1998:15:06:16/offices/0SERS/IDEA/whatsnew.html 

syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 15 : 06 : 36/ of f ices /OSERS /IDEA/ index . html 
syrul82- 

084 . syr . edu [ 10 /Nov/ 1998 : 15 : 07 : 08 /of f ices /OSERS/ IDEA/geninfo . html 
syrul82- 

084 .syr. edu [10 /Nov/ 1998: 15:07: 33 /off ices /OSERS/IDEA/summary. html 

syrul82- 

084.syr.edu[10/Nov/1998:15:07:50/offices/0SERS/IDEA/q_and_a.html 

syrul82- 

084 .syr. edu [10/Nov/ 1998: 15: 08: 44 /off ices/OSERS/IDEA/overview. html 
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Appendix V-3 Log Files, Group 2 



syrul82- 

084 . syr.edu [ lO/Nov/1998 : 15 : 10 : 22/of f ices/OSERS/OSEP/Programs/TAD_IDEA. h 
tml 

syrul82-084.syr.edu[10/Nov/1998:15:13:18/offices/OBEMLA/aboutus.html 
syrul82-084 . syr . edu [ lO/Nov/1998 : 15 : 16 : 12/pubs/EDInitiatives/98/98-10- 
29.html 

syrul82-084 . syr . edu [ 10 /Nov/ 19 98 : 15 : 16 : 4 9/f aqs . html 
syrul82-084 . syr. edu [ 10 /Nov/ 1998 : 15 : 17 : 44 /Picks/ index. html 
syrul82-084 . syr . edu [ lO/Nov/1998 : 15 : 21 : 08 /EdRes/ index. html 
syrul82-084 .syr. edu [lO/Nov/1998: 15:21 :33/EdRes/EdFed/Star. html 
syrul82-08 4 . syr . edu [ 10 /Nov/ 1998 : 15 : 22 : 37 /prog_info/StarSchools / 
syrul82- 

084 . syr . edu [ 10 /Nov/ 1998 : 15 : 22 : 46/prog_info/StarSchools/whatis . html 
syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 15 : 26 : 25 /programs . html 
syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 15 : 27 : 04 /Welcome /overview . html 
syrul82-084 . syr . edu [ lO/Nov/1998 : 15 : 30 : 20 /funding . html 
syrul82-08 4 . syr . edu [ 10 /Nov/ 1998 : 15 : 30 : 39/pubs/KnowAbtGrants/ 
syrul82-084 . syr . edu [ 10 /Nov/ 1998 : 15 : 31 : 31 /pubs /index. html 
syrul82-08 6 . syr . edu [ 10 /Nov/ 1998 : 14 : 34 : 10/ 

syrul82-08 6 . syr.edu [10 /Nov/ 1998 : 15 : 16 : 25 /pubs/ EDInitiatives/ 98/ 98-10- 
29.html 
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Appendix V-4, WebMac Scoring Chart 
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Appendix V-4, WebMac Scoring Chart 



WebMAC Website Motivational Analysis Checklist (3.1) Page 7 



Plotting Template 



Plot the score for V along the Value continuum; plot the score for XS along the 
Expectation for Success continuum. Then, draw straight lines to their point of 
intersection. Good websites will have both scores that fall in the upper right 
quadrant. An awesome website will have scores that fall in the extreme upper 
right quadrant. 
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Appendix V-5, Buttons Method Forms 
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Appendix V-5, Buttons Method Forms 

EDEVAL BUTTONS EVALUATION 
November 1998 

Interviewer instructions for conducting buttons analysis. 

1 . Prepare page packets 

2. See A under participant instructions below . Thank the participants for coming, restate 
the purpose of the study and provide confidentiality statement, and ask if there are any 
questions. 

3. Pass out demographics questionnaire 

4. Read instructions aloud (see B below) 

5. Pass out page packets and packet evaluation handout 

6. Stop the participants after 1 hour has passed. 

7. Ask participants if they have any questions. 

8. Begin the focus group session (see D below). 

9. Collect all materials. 
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Participant Instructions* 



Ask participants to excuse you for reading these instruction verbatim. While this may be 
a bit awkward, it will ensure that we do not mix up any of the instructions. 

A. Introduction to website evaluation 

Thank you for agreeing to participate in this study. The purpose of this study is to 
evaluate the Department of Education websites. Your participation is extremely helpful 
as it allows us to observe how an “interested” user interacts with the Department of 
Education website. 

We will be using a variety of data collection techniques in this session. First we will ask 
you to fill out a demographics questionnaire. Next we will ask you to examine and 
evaluate some printed out web pages. Finally, we will have a group discussion regarding 
the web pages you have evaluated. 

We want you to understand that we are evaluating the website, not you, or your use of the 
website. In this regard, it is not really possible for you to make a mistake or do 
something wrong. To the extent possible, we hope that you conduct your visit just as you 
normally would if you were exploring on your own at home or work. 

Although we don’t anticipate confidentiality being an issue, be assured that all notes and 
survey forms will only be accessed by the survey team and will be destroyed after the 
study is completed. Study findings may be retained but will not include personal 
identifiers. 

Are you still willing to participate in this study? 

Do you have any questions so far? 

Now we would like you to fill out the demographic portion of the survey. 

<hand out demographic sheet> 

Do you have any questions? 

B. Buttons Evaluation 

This data collection activity will consist of three parts; first, writing a brief written 
description of and reaction to webpage handouts; second, hand editing printed out copies 
of the webpages, and finally a short group discussion. 
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<hand out buttons evaluation worksheet; hand out eight page packets> 

Please locate each of the eight packets of paper. Each packet represents the contents of a 
webpage on the Department of Education website, stripped of all its identifying and 
indexing information. We would like you to thoroughly read the content of each of these 

pages. 

After reading the printout of each page, we would then like you to type a short paragraph 
on the computer which includes the following: 

1 . A brief description of the page’s contents as you understand them 

2. A brief reaction to the page. This reaction can include all of the following: your 
ability to understand the contents of the page, your emotional reaction to content on 
the page, questions you may have regarding the content of the page. Your reaction 
should also include anything else which the page makes you think of - positive or 
negative. I would like to read you a few sample reactions: 

“This is pretty clearly a ‘scope and purpose’ of the website type of page. 

This page should be near the top. Some of these links should probably 
even be on the main page of the site. The ‘guide to., links especially. 

Many of these links do not belong together on one page. The dogs report 
belongs with other links to dogs. There should be an entire set of pages 
concerning cats” 

“This page is a mess. I don’t have the slightest idea what the logic was in 
putting this page together. The few links are all dogs of one type or 
another, but then it gets into cats, birds and small rodents. All of these 
areas need their own page and to be expanded 

Please take the time to type your description and reaction to each page immediately after 
you have finished reading it. 

After you have written your response to the page, we would like you to edit the page with 
a pen. We would like you to: 

1. Circle those parts of the page you have a difficult time understanding. 

2. Cross out any content which you think is inappropriate for the page. 

I have included an example of an edited page on your handout. <point out example of 
marked web page> 

Please edit each page as soon as you finish writing your reaction to it. 

After you have finished examining, summa::izing/reacting to and editing all the page 
packets, we will hold a brief discussion to share our experiences with these Department 
of Education webpages. 
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To summarize. We will give you a packet of web page contents with all identifying and 
indexing material removed. Please read the first page, write the summarization/reaction 
to it and then edit it. After you have finished the first page, move on to the second. 
Please read, react to and edit the second. After you have finished the second, move on to 
the third - and so on, until you have finished all the packets. After everyone has 
completed their packets, we will hold a brief group discussion. 



C. Focus Group 

The purpose of this focus group is to gather information from you regarding your 
experiences with the Department of Education website. 

Which pages did you find the most problematic? 

What within the pages made them problematic? 

How could the Department improve these pages? 

Thank you very much for participating. The Department of Education will use your input 
to help redesign several aspects of their site. 
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Appendix V-6, Full Text of Buttons Reactions 
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Participants reactions vary partially due to the order in which they viewed the pages^ The 
researchers rotated the order in which participants evaluated each of the pages m order to 
ensure that each page received equivalent evaluatory effort. Thus, some participants 
reviewed each page at the beginning of the session, and some reviewed each page at the 
end of the session. Participant fatigue probably impacted some later page reactions. The 
researchers have not separated out or marked these later reactions. 



ED Offices and Budget 

1 Seems pretty straight forward. Its a good place to start. I don't understand the title Ed 
Web Awards. 

2. Description; The “About US” section. Important information is missing (e.g. Vision, 
standards, who we are, how to contact us, etc). I do not see relationship between 

“Company Info” and budget? 

Reaction; I did not like it. 10 links stated without description. Those bullets are ugly. 

3. page content; a basic directory of the U.S department of education 

reactions; clear and nothing very new , not much to react to. - would not change anything 

4. Brief Description; This is an introduction page to the DOE. 

Brief Reaction; Nice. Why do we have ED Web Awards in there? 

5. Overall, the information provided is giving me a clear understanding as to where I am. 
It is primarily the order in which it appears on the page that is weak. It also appears that 
information pertinent to this theme is missing, as well as, some of the information 
provided is not relevant to what appears to be the theme. Not too bad, but does need 
work or further development. Provide descriptives for the links. 

1 . administrative information about department of education, it looks line (the left side) 
introduction of U.S. department of ED. 

2. If the amount of national education goals is too large, and it is very important, what 
about just saying in this page instead of making hyperlink? In the sixth line, th^^on 
refers to, maybe, acrobat, but do you think how many people know about this? They 
might just try to click this icon. No good for their search. 

ED web Award is typed the biggest letter. It means it is most important in this page. 

7. Page starts out with a link to the Department’s Mission and National Education 
Goals. This is good. Maybe they should be described right on this page rather than 
having to go to another page to read them. No descriptors hampers efficiency because 
users may end up going to links that are not appropriate to their needs. This page looks 
like a table of contents to the web site. Not sure why the last link is listed and not 
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something else. Rliyme or reason to why some links are listed and some are not is 
unclear. 



Other Sites 

1. This is another list of links and links to outside sources. Again, without clanfying 
information, it is hard to know why these are organized the way they are. The disclaimer 
in the middle box is annoying. A lot of uninterrupted text is boring as well. 

2. Educational resource organization directory, that is, listings of nation-wide 
educational information sites 

Front part is just listing! Isn’t it necessary to classify these resources? For example, a. 
financial funding source .b. educational technology-related information. . . 

3. I was initially drawn to the box in the middle of the page and after reading it wondered 
why it was there. Not necessarily questioning the statement, but questioning the 
olacement of the statement. Again, the page appears to have some continuity, but then 
goes off in some other direction. The “state” agencies links should all be on a separate 
page and the last six links seem to be just thrown on there. Where is the theme? 

4. Brief Description; Various centers, agencies, web sites, projects, sites 

Brief Reaction; How are these affiliated to DOE? What is the function of each? Is there 
any common theme? What is this BIG disclaimer that eats up half of the page and 
separates information? TOO MUCH INFORMATION. I wouldn’t even know where to 
look if I needed specific information. These have to be annotated by mission, focus, 
purpose, or some theme. 

5. page content; I am a bit confused from it, I think it is about special ed and 
rehabilitation. I am not sure what the page would be called. It seems like there is a mix 
of information provided. On page 2 of page B for example. It seems like it is about 

Rehabilitation and special ed., but that is a field I am doing my ms. . t rv ^ 

Reactions; A bit confusing, needs more headings; does not look like a parent site ; I liked 

the box. To much on one page with not enough order would like all the libraries 

information in one place ang a briefing of the information in the categones; all the family 
and parent information in one area ext.. 

6. Description; Confusing information. I do not know if this is a “search page or 
“links” page. Not even a small piece of text introducing what is this all about. 

Reaction; I did not like it. Why a section without narrative and then at the end some 
narrative. The organization it is not comprehensive and fnendly. 
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7. This information does not seem to be organized in any particular order to make it 
easy to use or find. As a beginner in using the web it shows me information but l am not 
sure what to do with it. The box in the middle of the page is very confusing. I think so 
what. On page 2 what is a support institution? This just seems to be a mixture of 
information that rambles on. 

Research and Statistics 

1. Again there seems to be no logic to this page. I'm sure there is but its not mine. 
Where it says search eric where would I even begin. ( 900'000 + research articles) 



2. Description: I guess that the page is a “links” page, where a bunch of websites 
somewhat related with the organization are posted here to give users an idea of what 
organizations are related to this. 

Reaction: Each time I see a “links” page without annotations, or at least suggestions 
drives me crazy. 

3. Page content: a list of all the research organizations relating to education, 
reactions: very clear; well put together, would want to see such a page if I am looking for 
statistics when I am looking for information 

4. Brief Description: 

Statistics 

Assessment 

Clearinghouses 

National and Regional Labs 

Search Engines? 

Evaluation Efforts 

Brief Reaction: I am tired but also overwhelmed by so many institutions. I have no clue 
what service each provides from this page. I also don’t know why they are all on the 
same page and how they are affiliated with DOE based on the info provided. I 
desperately need some overarching orgamzational structure that is annotated and 
succinct. 

5. Clear theme, but no descriptions to the links. Some links are well placed and are 
comfortable to follow, but then it happens again where all of a sudden there are a group 
of links thrown at the bottom of the page. Discarded as they have no place in this site. 

Poor links! 



1. Information about educational statistics and measurement(like testing) 
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2. Well-organized and neat design. First level bullet and second level bullet make users 
know about the structure and relationship of each title. What's the major difference of 
Italic letter? Does it mean some sort of edition like book, or newsletter? If the ' Digest of 
educational.." and "The condition of.." are under the statistical publishing, you should 
narrow the space between these two lines and change the third level bullet to another 
smaller one. 

7. This page is like looking at a telephone directory. The yellow pages are much more 
interesting and at lease in alphabetical order. I imagine I would have to do a search to 
actually find one of these links if I couldn’t find it on the page because there doesn’t 
seem to be a rationale for why these are organized the way they are. Some items are 
categorized like statistical info and assessment info which is more helpful. 

Programs and Services 

1 This is a page of links with short summary descriptors. Descriptions are ok and may 
be useful in helping reader to access information. I prefer links with descriptors than 
without. Font size on all pages is small (looks like 10 point) and I would rather read 20 
point font because I read quite a bit during my regular work day. Abbreviations are used 
after each title which is helpful in case abbrev. Are used without full name later. 

2 . 

1. First page of department of education? 

2. Simple! 

3. Not bad! I like the short descriptions for the links. A few of these links seem out of 
place, not necessarily in the same theme. They should be grouped and seperated with a 
heading. 

4. 

1. Brief Description: This page appears to be an index page of some sorts. Has sub- 
headings that consist of directories, catalog, policy guidance, plans and annual reports, 
guide to education programs. 

Brief Reaction: I think its ok now that I have figured out what is common to all these 
links. However, a brief overview of this page will guide the user to what this page is 
about. This page has too much information that need not be related. I see four categories 
a) directories, catalogs, and guides and b) rules and regulations, c) reports, and d) 
overviews to ED programs and services. 

5. page content: gides and resource information for ed progrms 

reactions: I liked the briefing under each heading. It saves the time form going into the 
sit, I don’t think it would have been as useful other wise. 




2.96 



Appendix 125 



6. Description: An annotate list of options. Maybe a repository of documents and 
program descriptions and services altogether. 

Reaction: I do not see a hierarchy, or organizational schema. Even though in this page 
the links are described in advance, and the layout is better, still confusing. Subtitles, 
bottoms, or colors might help. 

7. Good idea to bold titles but but not in any logical order. 

Student Financial Aid 

1. I think there should be a list of definitions of different types of agencies and 
terminology. There is a need for an index so you can more easily find the general 
categories you want This could be very much like a book index. 

2. Description: This page was a lot clear, without title or subtitles was very cle^ the 
purpose. To provide information and advising to people seeking students’ financial 
support. Every links have a short description, nice. 

Reaction: This page looks a lot better than the previous one. Did not impressed me 
neither depressed me. 

3. Brief Description: This page is about various type of financial aid, loans, and ED 

projects that deal vrith the process. 

Brief Reaction: I like this page but it is overwhelming. It has many resources on how 
the DOE provides you with these avenues to pursue financial aid. The part that I am not 
ok with is again the hierarchy in which sub-heading is presented. I see two main 
categories a) type of financial aid, b) tax credits and c) agencies or projects that provide 
directives and regulations for the process. Then from there they can lead one to the 
various sub-headings. 

Now that I have re-organized this more I look at my category A and see that there are still 
too many areas. I would like to sythesize the info more so we could have 3 categories a) 
Financial Aid b) Grants, and C) Loans or something like that.. 

4. 

1. Resource for financial support 

Title of fund or scholarship should be highlighted. Under the fourth bullet, I almost miss 
the hyperlink "direct loan web site" 

5. Information to li nks is in a listed format with short summary type descriptions. 
Nothing of interest pops out at this time. Might be more informative if it described what 
constituents would be interested in each link. Use of abbreviations without the official 
name is presuming the users know these in advance. 
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Funding Opportunities 

Due to researcher error, this page did not receive the same amount of participant attention 
as other pages. In order to correct for this, the researcher had two web design experts 
assess the page using the same protocol as the regular participants. The expert assessors 
comments follow: 

1. This page appears to be a list of federal sources of educational funding, for both 
research and the implementation of programs. They should add descriptions for all the 
links (i.e. why one might want this information). 

Lack of descriptions impedes understanding of web page. How are Federal Register 
documents, Ed General Administrative Regulations, ED Budget, Comprehensive School 
Reform Demonstration Program and Safe & Drug Free Schools Program links relevant? 

2. This page is all about Ed Grants, Funding and Budget information. But, the Safe & 
Drug-Free Schools link at the bottom of page doesn't seem to fit with those areas. 



President's and Secretary's Priorities 

1 . Lack of titles impedes understanding of web page. List and description of the 
Departments seven priorities would read better if they were in chronological order. I m 
unsure why they are described out of order. The links under each paragraph appear 
appropriate, but it is not clear whether they are appropriate without seeing them. Not 
sure if the page is written so that all users can understand it. If the audience is parents, 
more clarification is needed. 

2 . ... 

1 . Educational regulation and policy guideline for priorities 

Even though the summary of each goals is posted in the first part of this web page, I do 
not think it works because few people know it shows the contents beforehand. What is 
this for? Convenience to link? 

3. I was understanding the gist of the page as I began to read it and then in the middle of 
the fist page was apparently the basis or lead in for the information on the ^ ^ 
continued to read through the page, I felt most links were appropriate for the “Call to 
Action” point that was being made. However, I am not sure I was comfortable with me 
order in which these points were being made (no apparent sequence). As for the design, 
links relative to the “points” were good in list form and when a few of them were placed 
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in a paragraph format, it was out of sync. Good idea, but needs to be reformatted for 
continuity. 



4. Brief Description: The page provides users with information on how the DOE is 
sponsoring initiatives that address the presidents’ Educational Goals. It provides simple 
links to the various organizations and projects invo loved in carrying out these initiatives. 
Brief Reaction: Nicely done. They managed to put in a lot of information in an easy to 
synthesize maner that can be easily tied to the main purpose of the web page. 

Annotations of each link might be useful but not at the expensive of overwhelming the 
user. Also for every category, there are 7 + /- access points which is not taxing to the 
brain. 

5. Description: I am not sine about he content. May be is related to the school reform or 
some king of organizational objectives based on President Clinton’s “Call to Action”. 

Not very well layout. Numbers or bullets might help to differentiate one objective from 
other. The fonts are used without concern about the size, normal paragraphs one size, 
l inks other size, and so on. 

6. Secretary Riley message should have been put at the top of the page. I didn't know 
what I was reading or what the context was. The organization of this material is terrible! 
! Different type fonts inappropriate placement of information. 



News and Events 

1. A hodge-podge of information. I am getting tired of reading it . I think I would look 
for information somewhere else. All of the material on all pages is in a formatt the is 
frustrating to look at and read. I am not sure I could find anything I wanted. 

2. Description: Seems to me like an press section. News, reports, calendars, etc. Without 
categories, it is hard to say for sure what is this page all about. 

Reaction: I am not sure, but are all these pages created without any kind of graphics. I 
mean to reinforce the written language with the graphics language. People are so used to 
read newspapers, magazines, books, with heavily graphic content. 

3. Brief Description: Ok the first part is about community updates and that takes you to 
all sorts of information. All relevant I hope. Then there is a section in the center that 
does not seem to fit anywhere. All I understood from it was that it is about reports and 
announcements, letters, bulletins etc.. Then the third part is about the main page??? The 
links in there do not look f ke they merit making the MAIN page of the DOE! ! But then I 
am not an expert in that area. I thought I was done only to discover there is yet another 
page. . . of stuff totally unrelated to the top of this page. I try to make sense of the content 
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but it is overwhelming. . .too much detailed information covering such a range of topics 
from grants to conference, project events, training, etc... etc.. 

Brief Reaction; Well, This whole page, aside from the Main ED WEB part seems to be 
an index of various publications and press releases. Ok good, but there is too much in 
there that may not be immediately relevant if you are just trying to get a quick update 
from various publications (see editing). I really do not care for this page at all. As I 

read through most of these links take you, it seems, to lots more publications. For 
example ED press releases, town meetings, etc. . . why not just do a chronological update 
on these issues that matter and archive information by date. The second page is totally 
frustrating, overwhelming, and deals with a very different topic from publications and 
press releases. 

4. My initial thought was that was no rhyme or reason to the links. Certain groups of 
links appeared to be relevant, while others just appeared to be thrown down. Main ED 
Web is sticking out like a sore thumb. Where does it belong, possibly not on this page. 
Everything is a link with no descriptives to both the links and the page. Why is there a 
statement as where more info is without a link to that info. Very chaotic! 

5. 

1 . listings of programs and service. 

2. It is like melting pots. NCES, OPE, SFA are under the white house education press 
release? If not. What is the relationship between first part bullet and second part bullet? 
Isn't it too spacious between Main ED web and five second bullets group? The last ten 
first bullets can be classified just "the others"? What is the main subject of those things? 

6. Lists of links with no descriptors. No buttons to Homepage or a vehicle to e-mail 
questions and thoughts. Hard to tell if the links are appropriate without going to them. 
More abbreviations that may not be readily deciphered. 



Publications and Products 

1. No logical order. 

2. Description; Better, simple, and nice. Resources are presented with a short 
description. The user can easily review all of them, they would think that because the 
description is short the content is easy (personal assumption). 

Reaction; I did like it. The fonts are OK, the descriptions. The part at the end is just nice, 
is like “if you still want more see also this”. 
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3. page contents: the page is an education resource directory. It contained informative 
information such as data bases, resource directories for and ordering systems for 
publications and gides to the U>S department of education. 

reaction: I have a clear ability to understand the contents of the page. If I to look up 
something about education for research reasons or the department of education, I would 
hope to find, the information I see on the page. It gives me a path to start out on. 

4. Brief Description: This page is about various publications and databases, directories, 
research reports, and statistics including regulations and policies. 

Brief Reaction: Haven’t I just seen similar information on Page D? Too much to 
browse through. Again, I suggest further synthesis by categorizing this page into a) 
Directories, b) Databases, c) Reports or soemthing like that. The annotated information 
is useful. The later section (see also) is distracting and does not add any value to the web 
page. Either include it in one category or remove. More pages, less links, more short 
annotations. 

5. This has been the most concise page thus far. There is a brief description for the 
majority of the links which gives me an opportunity to know if that link has relevance to 
my research. However, I would like to see these in a more heirarchial order or possible 
grouped in more closely related topics. There are a few links that I thought “why are they 
here?”, otherwise this was clear and concise. 

6. updated version of first page of U.S. department of ED(if I compared this to page D)7 
Too many listings. Wouldn't it be nice to lessen the number of titles to six to seven at 

most? 

7. Another list of links. Doesn’t appear to have a rhyme or reason for the list. Not sure 

how else it could be presented to improve it. Titles seem to be ok. Not sure why the see 
also is listed on the bottom of the page and why those links are there. 
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Appendix V-7: Buttons Evaluation Page Images 
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Presidents and Secretary’s Priorities 



1 .All students will read independently and well by the end of 3rd grade. 

2. All students will master challenging mathematics, including the foundations 
of algebra and geometry, by the end of 8th grade. 

3. By 18 years of age, all students will be prepared for and able to afford college. 

4. All states and schools will have challenging and clear standards of 
achievement and accountability for all children, and effective 

strategies for reaching those standards. 

5. There will be a talented, dedicated and well-prepared teacher in every 
classroom. 

6. Every classroom will be coimected to the Internet by the year 2000 and all 
students will be technologically literate. 

7. Every school will be strong, safe, drug-free and disciplined. 



Secretary Riley and senior Department officials developed seven priorities (see 
Working Document 

and the Strategic Plan) for the Department, based on the "Call to Action" issued 
by the President in 

his State of the Union Address (February 4, 1997). These seven priorities are as 
follows... 



All students will read independently and well by the end of 3rd grade. 

America Reads Challenge 
The Reading Summit 

Class Size Reduction and Teacher Quality Initiative 
America Goes Back to School; Get Involved! 

Voluntary National Tests. 

Family Involvement 

National Assessment of Educational Progress (NAEP) Reading Reports 

All students will master challenging mathematics, including the foundations of 
algebra and 

geometry, by the end of 8th grade. 

Visit our Math Initiative Page where you will find information such as 
speeches, fact 

sheets, publications, reports and the Third International Math and Science 
Study (TIMSS). 
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By 1 8 years of age, all students will be prepared for and able to afford college. 
(Point 8 in President Clinton’s “Call to Action” 



America's HOPE Scholarship 
Direct Loan Program 

Department of Education's Office of Postsecondary Education 
Project EASI 

Think College— Learn for a Lifetime 

All states and schools will have challenging and clear standards of achievement 
and 

accountability for all children, and effective strategies for reaching those 
standards (Point 1 in President Clinton’s “Call to Action”). 

Voluntary National Tests 
Goals 2000 
School-to-Work 

Improving America's Schools Act of 1994 — Elementary and Secondary 
Education Act 

Comprehensive School Reform Demonstration Program 

National Assessment of Educational Progress (NAEP) State-by-State Results 

There will be a talented, dedicated and well-prepared teacher in every classroom. 
(Point 2 in President Clinton’s “Call to Action”) 

Visit our Teachers Web Page where you will find information on attracting 
and preparing 

teachers, guides for teachers, teacher leadership, research and studies, ERIC 
Digests, and 

other resources. 

Every classroom will be connected to the Internet by the year 2000 and all 
students will be 

technologically literate. (Point 10 in President Clinton’s “Call to Action”) 

U.S. Department of Education Technology Initiatives 

The Mid-continent Regional Educational Laboratory (McREL) 

North Central Regional Educational Laboratory 
Regional Technology in Education Consortia (R*TEC) 

Every school will be strong, safe, drug-free and disciplined. . (Point 7 in President 
Clinton’s “Call to Action”) 

School Construction and Design 
Safe & Drug Free Schools Program 
US Charter Schools 
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Flexibility and Waivers 

Improving America's Schools Act of 1994 — Elementary and Secondary 
Education Act 

The Partnership for Family Involvement in Education supports the seven priorities 
through family and community involvement in children's learning. As members of 
the Partnership, thousands of 

family-school partners, employers, community organizations, and religious 
groups work together to 

help all children learn to high academic standards. 



Funding Opportunities 



Grants and Contracts Information 
Federal Rf^^ixter documents 

1999 Forecast of Funding Oppominities under ED Direct Grant Programs and 
Fellowship Programs (1 1/3/98) (also av^lable in PDF tonnat 1.09Mb)^^^^. 

What Should 7 Know about E D Grants ■ 

F.D General Administrative Regul ations (EDGAR) 

Guide to ED Programs 
ED Budget 



o Rilinoiuil Ediionrinn and Minoiitv Languages (OBEMLAl Fu nding 
Qnnnminicies 

O GniTinrehensivK School Ret-'orm Demonstration Program 
I.' Fundin? for Postsecondarv Institutions 
o Migrant Education Grant Information 

o Rehabilitation Services Administration CRSA^ Pro gram Application 
Kits 

O Safe & Dru?-Free Schools Proeram 
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Student Financial Assistance 



. User-friendly publications (such as the popular Student Guide to Financial Aid 
and F,mdir. Ynnr F.diicaiion) describing federal grant, loan, and work-study 

programs. 



. Annlication for Federal Stud c>nr AidfFAFSA^ - electronic versions of the 

FAFSA that you can use to submit your completed application form to ED 
through the web, by modem, or by mail. (If you have technical questions about 
the electronic FAFSA, please call 1-800-801-0576). 

• TiMr» TV Sch<inl rode Search , a convenient way to look up codes for the colleges 
that you are considering, so that you can enter them in Section H of the FAFSA. 



A special low interest rate, currently 7.467o, is available for a limited time Jo 
eli^Tible borrowers who consolidate their outstanding student loans through toe 
Direct Loan program, if their application is received no later than January 31, 
1999 (This interest rate may also be available through some 
aovemment-guaranteed lenders, although they are not required to offer the low 
rate.) For more information, call 1-800-557-7392 or check out the __irect — pm 
Web Site at http://www.ed.gov/DirectLoan. 



. Hnne Scholarship and Life time Leaming.Tax Credits which provides aU the 
latest information on the new tax credits for college costs. 

. The Guide to Defniilted Student Loans , designed to help borrowers resolve, 
default situations for loans that have gone into default. 



• Other higher education resources , including addresses and links to <;rnre guarantee 
agencies . 



• OPE Policy Initiatives and Program Data and Information for Financial Aid 
Professionals which provide officisl policy and regulatory directives or 
postsecondary education institutions. 



Prni.rrT=A.ST. a coUaborative initiative led by ED and ta partner “ * 
education community. Project EASI focuses on ways to streamline the fman 

aid process for our customers. 
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Research & Statistics 



• vfational Cenrer for Education Statistics fNCESi including 

t 

O Fdiirntional Statistics F inance Center 
o Frpqiipnrlv Asked Statistic al Questions (uses frames) 
o Statistical n:ira and Surveys 
o Statistical Indicators. 
o Statistical Piililications including ... 

■ Divpst nf Education S tatistics ( 1 997 ^ 

■ Th,> CnnAitinn nf Educa non ( \992>) 

o Third International Mathem atics and Science Stuciv fmtS^ 



• National .‘Assessment of Educatio nal Progress (N' .AEPl 



o National Results 
O State, hv State Re.sults 
o Results over Time 
o Summar'/ Data Tables 
o Special Interest Reooits 



• National Asses.sment Governing Board (NAGB.) 



• Educational Resources I n formation Center fERIQ . 

. Eisenhower National Math/S ^enre Clearinghouse and Regional Consort L, 

• Fund for the Improvement o f Postsecondarv Education 

• Narional Institute for Disabiliti e s & Rehabilitation Research (NIDR Rl 

• National Lihmrv of Education (NLE1 

• Office of Ednrntional Research & Imprnvement fOERU 



308 



Reiiional Educational Lab ordtones 
Nnrionnl Research & Dev >-lnpmear Centers 
Field-initinted Studies 

Ncitioncil Educotinnal Researc h Policy and Priorities Board 
National Center Kor Resenrch in Vocational Education rNCRVEl 



O Nnrinnnl Institntp on Studont Achievement. CuiTic iilum. and 
Assessment 

o National Institute on the E d ucation of At-Risk Students 
O Nntinnnl Tnstjnifp on Early Childhood De velopment and Educatipii 
O National Tnstimtn on Educational Governance, Fin ance, 
Policy-Making, and Manag ement 
o National institute on Postseconda Pv' Education. Libiaries and 

r.ifelong Learning 



• Collections of Research Svnthese.s 

• Education Research and Practice - R eports and Studies 

• ORRf Butlerin - quarterly newsletter 

• Program Evaluation 

• Research Priorities Plan 

• Search ERIC - bibliographic database of 900,000-r research articles, reports, 
conference papers, instructional materials, and other materials (updated mon^ly) 

. Search ERIC Digests - 1 ,700-r short reports on current education topics (updated 

quarterly) 



News & Events 



• Community Undate 

• EDInfa InFoitmuion Service/Listserv 

• ED Iniiiative.s 

• ED Press Releases 

• ED Publications Published within the Last 90 Davs 

• Satellite Town Meetings 

• Speeches and Testimony 

• Updates on Budget and Activities 

• White House Education Press Releases'and Statements 



o National Center for Education Statistics (NCES) — Recently 
Released Reports and Data Products 
o Office of Postsecondarv Education (OPEl — News and Official 
Announcements from the Office of the Assistant Secretan' 
o Student Financial Assistance (SFA) -- Dear Colleague Letters . 
Official Electronic Announcements, and Direct Loan Bulletins 



• Main ED Web 



° National Center for Education Statistics rN'CESl 
° National Library of Education 
o Office of Vocational and Adult Education (’OVAEl 
° Safe and Drug-Free Schools News Update page 
.o School-to-Work CSTW^ 




F^’diirot R^?is cer dociuTicnts 

FY TQQ9 Forprnst oF Funding On p nminicies unrier ED Direct 



Grant Programs and 






FRlln\vs;hin Pro«Trams ( 1 1/3/98) (also available in PDF foi-mat -- 1 ,09Mb) 




GranLs and Contracts Information 

More Info about Funding Opportunities in ED Ofrices 

o Rehabilitation Services Administration IRS A) Program Application 
Kits 



• 1998 Regional Conference.s on Imorovinu America's Schools 

• Calendar of Education-related Conferences (at ACCESS ERIC) 

• Calendar of N'TDRR Project Event.^ (National Institute for Disabilities and 
Rehabilitadon Research) 

• Disability-Related Conference Database (National Information Center for 
Children and Youth with Disabilities — NICHCY) 

• National Clearinghouse on Bilingual Education fNCBE t Conference and 
Meetings Calendar — 1 998 

• National Institute for Literacy Calendar of Events 

• School-to-Work Calendar of Event.s 

• SFAP Training Update Calendar for Financial Aid Professionals (Student 
Financial Assistance Programs) 

• Satellite Town Meeting 

• Vocational Education Calendai' of Events (National Center for Research in 



Vocational Education) 



Programs & Services 



• Overview of ED Programs and Services — This introduction to the Department's 
primary areas of responsibility highlights key programs and initiatives and provides numerous 
links to additional information. 

I 

• Guide to U S. Departinpnt of Education Programs - In order to carry out its 
mandate, the Department administers over 200/programs. The Guide provides a concise 
description of each program, identifies who may apply, and gives the name and telephone number 
of the Department office to contact for more information. The Guide is updated monthly. 

• Education Resource Onranizations Direct or^' (EROD) -- The Directory is 
intended to help you identify and contact organizations that provide information and assistance on 
a broad range of education-related topics. The Directory includes information on more than 2,000 
national, regional, and state organizations and is constandy being updated and expanded. In many 
cases you can link direcdy to the organization s home page. 

• Catalog of Federal Dompstic .A.ssistance ( CFD.AI (at GSA) - The CFDA is a 
comprehensive listing of all Federal government programs - not just those from the Department 
- that give out money or other forms of assistance. The CFDA is published every year in June by 
the General Services Administration (GSA), with an update published around December. 

• Biennial Evaluation Renort - The FY 1995-96 Biennial Evaluation Report provides 
available information on the purpose, funding, target population, services, administration, 
effectiveness, management improvement strategies, and sources of information for 154 programs 
administered by the Department during FY 1995-1996. Also Available; Pi<;pni^ L E^'^tiatiQ ri 
Renort l99?-94 . 

• Legislation. Regulations, and Policy Guidance - Our growing collection of 
legislation, regulations, and policy guidance pertaining to ED programs and activities includes 
links to full text documents on ED's web as well as useful resources elsewhere on the Internet. 

• ED Strategic Plans and Annual Reports 




Publications & Products 



• ED Pubs On-Line Ordering vSvstem -- to identify and order current U.S. Department of 
Education publications and products. 

• Database of ED Publications in ERIC -- a searchable bibliographic database of more 
than 20,000 publications produced or funded by ED since 1980. 

• Newsletters and .Tournals from the Department -- periodical publications featuring 
news about Department initiatives and programs, upcoming events, new publications and services, 
and the latest research findings and model programs. 

I 

• Guides to the U.S. Department of Education — general overviews of the Department 
and road maps to its programs and offices. 

• Resource Directories — catalogs and collections of information on a wide range of 
educational programs, events, and organizational sources of assistance. 

• Collections of Research Syntheses — concise, research-based synopses and literature 
reviews of major educational topics. 

• Publications for Parents — electronic versions of popular pamphlets and brochures 
designed to address parents’ concerns about their children's education. 

• Education Statistics -- an extensive list of stari.stical nuhlicarion,s. survey data s ets.> and 
topical analyses produced by the National Center for Education Statistics (NCES) to report the 
condition and progress of education. 

• Educational Re.search and Practice - Reports and Studies - comprehensive, 
timely analyses of major educational topics such as systemic reform, the relationship between 
time and learning, and research-informed transformation of instruction. 

• ED Strategic Plans and Annual Report.S -- strategic plans, annual and semiannual 
reports, and other administrative reports and plans describing the activities of U.S. Department of 
Education offices and programs. 

• Legislation. Regulations, and Policy Guidance — full text documents on ED's web as 
well as useful resources elsewhere on the Internet pertaining to ED programs and activities. 

See also; 

o President's & Secretary's Priorities 
o Program Evaluation ~ReDorts 
o Civil Rights in Education Publications 

° -Student Financial Aia ProPocc- — 
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ED Offices & Budget 



Mission 

Nacional Education Goals 



RD Organizational StP irniro and Offices [Text-onlv version] 
Where ED k T.ncaced-HeadQiiarte rs and Regional Offices 
Find a Pei-snn/Phone Directory 
I99R ED Qi-ynnizational Directory 






ED Budget 

Overview of ED Programs an d Activities 
Guide to ED Erogranns 



Other Sites 



o Search anv or all ED-ftinde d web sites 
o Comprehensive Regional Assistanc e Centers 
o Educational Resources Information Cente r rERICi and Other 
Clearinghouses 

o Eisenhower National Clearinghouse a nd Regional Consortia 
o National Research and Developm ent Centers 
o Regional Educational Laboratories 
o Regional Technology in Education Consortia (R"*TEC) 
o Star Schools Program Sites 
o Technology Innovation Challenge Grant Projects 
o Other ED-Supported vSites 

Tlie categories below contain hypertext links or pointers to inionnation created and 
maintained by oilier public and'privatc organizations. These links and pointers are provided 
for the user's convenience. Tlie U.S. Department of Education does not control or guarantee 
the accuracy, relevance, timeliness, or completeness of this outside information. Further, 
the inclusion of links or pointers to particular items in hyperie.xt is not intended to reflect 
their importance, nor is it intended to endorse any views e.xprcssed. or products or serv'ices 
offered, on these outside sites, or the organizations sponsoring the sites. 



O Other Federal Agenev Educational Resources 
o Federal Government Internet Librar y Resources 
o General Guides to Government Ttitgmet Resources 



o State Arts Agencies 
o State Directors of Adult Education 
o State Directors of Special Education 
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o Stnre Diret'rni's of Vocational -Technical Education 
o State Edurarion Agencies 
o State Guarancv ^vgericies 
o State Higlier Education Ai? encies 
o State Literacy Resource Centers 

O State Parent Training and Infonnation Center ( Disabilities) 
o State Tech Prep Coordinators 
o State Technology Web Sites a nd Contacts 

O the PHiirntion Resource Organiz ations Directory ( EROPl for 
additional state and regional sources of uiformation and assistance. 

Educational Institutions and E d ucation Support Institutions -^including lists 
of Internet sites for colleges and universities, K-12 schools, school districts, and 
state departments of education. 

Libraries - including lists of Internet-accessible library catalogs and services. 



Educational As.sociations and Orga nizations 

Curricular Resources and Net^vol'king Protects Internet-based networking 
projects and educational materials for teachers and students. 

General Catniovs and Subie j^ Trees on Education 
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Appendix V-8, WebMAC Analysis Template, Group 1 
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Appendix V-8, WebMAC Analysis Template, Group 1 



WebMAC Website Motivational Analysis Checklist (3.1) Page 7 



Plotting Template 



Plot the score for V along the Value continuum; plot the score for XS along the 
Expectation for Success continuum. Then, draw straight lines to their point of 
intersection. Good websites will have both scores that fall in the upper right 
quadrant. An awesome website will have scores that fall in the extreme upper 
right quadrant. 



High Expectation for Success 



/ 



Low Value 
0-1 



Average to hi^h for Value/ Averof^c to hi;^h for Vaiuc/^ ' 
Below average to low far Average to high for . * 

Expectation for Success 



10 



20 



50- 



40- 



Expectation fory ' 

Success / 

/ 

/ 

/ Good/'V />.:•//» Wnc 
. ' and 

lor Snrrvss 



Awesome 

- 



’veiDSiTe! 






/ 



/ 



/ 



Averagti lor borii Vet Inc 
. _ and Ji\:.H’rtauonfor 
. Succl'Ss 

40 y 

-1-30 . M i l l l-t-1- 






I I I -1- 60 



High Value 



20 - 



10 - 



Below average to Value/ 
Below average to low for 
Expectation for Success 



Average to high for Value/ 
Below average to low for \ 
Expectation for Success 1 



0 
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Appendix V-9, WebMAC Analysis Template, Group 2 
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Appendix V-9, WebMAC Analysis Template, Group 2 



WebMAC Website Motivational Analysis Checklist (3.1) Page 7 




Plot the score for V along the Value continuum; plot the score for XS along the 
Expectation for Success continuum. Then, draw straight lines to their point of 
intersection. Good websites will have both scores that fall in the upper right 
quadrant. An awesome website will have scores that fall in the extreme upper 
right quadrant. 



Low Value 
04 



High Expectation for Success 



/ 



/ 






Average to hi for Value! gQ Ai'err/yc to hi'^lt for Valtie/^ 7 

. Averar^e to hi^h for ' 



/ 
A 



Below average to low for 
Expectation for lUitccess 



50- 



40- 



10 20 
1 I I I 1 I 1 I II 



I'xpcctation for^ 
Success / 



Awesome 

Website! 



^ CtoocI hnth Vainc 
. ^ and Eyj'cciatiofi 
for 



/ 






Avcni^c for botli \taiuc 
_ - and Experuuion for 
..Succe^ 

^ 40 

30 - 1- 1 1 Ilf 1 ! 



/ 



50 



High Value 
■60 



20 - 



10 - 



Below average to low for 
Expectation for Success 



Below average lo Value/ . - Average to high for Value! 



Below average to low for 
E'tpeaaiion for Success 
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Appendix V-10 WebMAC Analysis Template, Group 1 and 2 

Compared 
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Appendix V-IO WebMAC Analysis Template, Group 1 and 2 Compared 



\A/cbAAAC Website Motivational Analysis Checklist (3.1) Page 7 



PlottingjTetnplate 



Plot the score for V along the Value continuum; plot the score for XS along the 
Expectation for Success continuum. Then, draw straight lines to their point of 
intersection. Good websites will have both scores that fail in the upper right 
quadrant. An awesome website will have scores that fall in the extreme upper 
right quadrant. 



High Expectation for Success 









Low Value 
04 



Average ro hif*hfor Value/ ^ Avera'^e to hi^hfor Value/ / /\ VVS50/77S 
Below average to low for Averaf^e to hi^h for 

Efpeaatioii for Success | Expectation for^ '' Wohsitd 

Success / 



10 



20 



50- 



ami ti kx • tat ion 
Jor Sm cess 



y 



4o'r 

....Avenige tor both Value / 
^ - - atid Exi'vriaiion for 

40 / 50 



4-30-1- f l- 



20 - 



43 2 2 

10 - 

Below average to Value/ . - Avera.^e to high for Vahid 
Below average to low for - - Below average to low for 



Expectation for Success - - 



Expectation for Success 



/ 



/ 



High Value 
60 
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Appendix V-1 1, OESE Expert Evaluation Materials 
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Appendix V-11, OESE Expert Evaluation Materials 

U.S. DEPARTMENT OF EDUCATION WEBSITE EVALUATION PROJECT 

Please evaluate each of the subsites within the OESE website on the following five 
criteria. These criteria are merely presented as a guideline. Feel free to include 
evaluative comments that do not seem to fall into one of the criteria. 

Overview, scope, mission, liability, copyright if appropriate 
Aesthetics, consistency, and appropriateness of content 
Minim al user skills required, clear indicators/links, easy to 
understand terminology, link provided to appropriate homepages 
(OESE, ED), obvious and easy to understand finding aids 
No dead end links, outdated information, incomplete pages 
Obvious and easy mechanisms to provide feedback, obvious and 
easy to understand help features 

Make comments about the site on the on-line form currently open on your PC desktop. 

Given the number of pages we would like you to look at, it will not be possible for you to 
spend more than 10 minutes on each subsite. You may only be able to make one or two 
comments per criteria per page. We will alert you after every 10 minutes has passed. 
These time alerts should only act as reminders of the passage of time and the need to 
evaluate all eight subsites. Feel free to move at your own pace. 

Subsites: Compensatory Education (CEP) 

Comprehensive School Reform Demonstration (CSRD) 

Goals 2000 
Impact Aid 

Office of Indian Education 
Office of Migrant Education 
Safe and Drug Free Schools 
School Improvement 
Home Page/General 



Orientation 

Design 

Navigation 

Quality 

Customer Service 
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Appendix V-12, OESE Expert Evaluation Materials 
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Appendix V-12, OESE Expert Evaluation Materials 
Orientation 

Design 

Navigation 

Quality 

Customer Service 
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Appendix V-13, OESE Website Evaluation By Subsection of 

Website 



o 

ERIC 
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Appendix V-13, OESE Website Evaluation By Subsection of Website 

As explained earlier in the methods section, due to the breadth of evaluatory comments 
sought, each of the four evaluators did not have time to complete an evaluation of each 
subsection. The researchers rotated the list of assigned subsections for each evaluator 
however, to guarantee that at least one evaluator would look at every subsection. 

The comments of the evaluators regarding each subsection follow. Some subsections 
contain one set of comments, others contain two or three sets of comments. The 
researchers have not edited these in any way. 

Compensatory Education (CEP) 

Two evaluators evaluated the CEP subsection. Their comments follow arranged by 
evaluation criteria: orientation, design, navigation, quality and customer service 

Orientation 

This site is much better organized in relation to the other sites I looked at. At least there is a kind of overall 
“theme” that follows from page to page. 

Design 

The hand pictures are more or less interesting and clear to follow .. good job on putting the Imks at the top 
and bottom of the page .. I’d also think about having some consistent margins on either side of the page to 
make it more readable , ie: 

«^ABLE BORDER=0> 

«^TR> 

<TD WEDTH=”10%”x/TD> 

<TD> 

page data here 
</TD> 

WEDTH=”10%”x/TD> 

<A-ABLE> 

That would provide nice clean margins. 

Navigation 

Fine .. again with the hands. 

Quality 

Good again .. everything nicely dated and no missing links I could fine .. someone NEEDS to proof and 
edit these pages. Spelling errors galore and grarmnar ones as well. 

Customer Service 

There’s an email address and FAQs .. FAQs are always good. 

Overall 

- Not enough orientation to where l am at within the site. Did not have much time to do this activity due to 
server problems, overall, looking at the three sites, there wasn’t much consistency in design layout among 
them given that they are all from the main source. 

Orientation 

The mission and scope are clearly stated. There is an introductory section to every page which covers the 
criteria of evaluation. 1 think that is the most important page. The way it is currently designed is very 



328 



Appendix 143 



useful. I would like to be able to reach everything on the web sites from these splash pages. 

Design 

Appealing in terms of use of text and graphics as pointers. Font size is too small if catering to an elderly 
audience. The use of pictures on each into page is not useful or informational but does perhaps send out a 
message about the services provided in that there is an element of caring as illustrated in the use of pictures 
of children and graphics appealing to children. 

Navigation • u • 

Very easy navigation. The text and graphic based menu is not overwhelming and is placed in obvious 
places that are hard to miss. Again slightly larger font for these navigation bars would help. 

Quality 

No apparent dead links. Information is sufficient but not overwhelming. 

Customer Service 

I recall on several pages a reference to contact names and email addresses and “where to get help’ sections. 



CSRD 

One evaluator evaluated the CSRD subsection. His comments follow. 

Orientation 

The splash page here is misleading because it does not contain enough information and is quite redundant 
because if you go to the home page most of the information is there so why add this level when one could 
just go directly to that page. I thought that overall this criteria is well addressed. 

Design 

Why do we have this little icon at the upper left hand comer that takes you to the home page. It is too 
email distracting and is not the only way to get to the home page. I would suggest removing it. I don’t 
thinlf the splash page is well designed and is certainly not consistent in style with the more formal looking 
web page. 

Navigation 

I found navigation to be easy from the home page but got more difficult and unorganized when it got to the 
second and third levels. A lot of work needs to be done on this level. 

Quality 

Given the short time available to do a thorough job, I would say that I did not find dead links, the 
information appeared current based on last date of modification. I did not see any incomplete pages. 
Customer Service 

I did not seem to fmd much related to customer service. 



Goals 2000 

Two evaluators completed an evaluation of the Goals 2000 subsection. 

Orientation 

The goals and missions are well stated. 

Design 

The splash page is a good introductory page about mission and scope but the page is not well organized in 
tenns of what is in the site. Embedding the links in the text is not helpful because it distracts the user. The 
links also appear to be at the same level of in^)ortance when I do not think they are. 

Navigation 

Once you go into the second level, a listing of links does not help much especially when they take up the 
whole page and there are so many of them. 

Quality 

very outdated (j^ 1997) 

Customer Service 

I did not see any. 
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Orientation 

It seems to me that the Goals2000 site needs to fill two major requirements - first, to infonn people what 
the program is and second to help potential grant applicants complete the application process. The site 
provides this information, although not in a way that is particularly helpful to users. 

Design 

The design of this part of the site is obviously supposed to be simple and straightforward - and it does not 
take a great deal of time to dl over slow connections. Unfortunately, the site reads more like someone 
simply took pre-existing text and pasted it in, adding random links where appropriate. The site needs to 
take advantage of web features like the ability to link within a document to pull out major points .. there 
should also be more help for the grant writers. 

Navigation 

The navigation is horrible. Since there is no **main-menu”, each of the subsequent pages in the site doesn’t 
link back to anything except the dept of ed main page, which isn’t useful. There is no logical “flow” to the 
site at all. 

Quality 

There doesn’t seem to be any missing links .. most, though not all, of the info is dated but some is from a 
while back. This is confusing to users because info from last year might still be current, but users have no 
way of knowing this. There needs to be expiration dates on the pages. 

Customer Service 

There isn’t any way for users to provide feedback, etc .. this should definitely be here to provide help for 
grant writers. 



Impact Aid 

One evaluator examined the Impact Aid subsection. 

Orientation 

This looks like a site that was meant to list different types of grants and which office/dept they come out of. 
Horrible, horrible interface. At least you can search grants. 

Design 

This design if horrible. I understand the basic idea of having the tier or tree approach for each dept .. but no 
user is going to understand what those numbers are imdemeath them. Obviously this data is in some kind 
of database already, so you’ve got to come up with a better way of using the info returned from the db. 
Navigation 

The navigation is not consistent The “back to navigator’’ button takes me some place I didn’t start firom. 
When you expand a “tree”, the page leaps around like crazy and some users won’t realize what happened. 
There are no instructions, either. 

Quality 

No problems here .. all the info is there, like I said before - it’s the presentation that’s flawed. 

Customer Service 

Nothing at all .. it’s very easy to get lost on this site .. and who do you email/call if you have a question? In 
the individ grants themselves there is contact info, but not for the whole site. 



Office of Indian Education 

<evaluator copied his comments from his Impact Aid evak*ation> 

Orientation 

This looks like a site that was meant to list different types of grants and which office/dept they come out of. 
Horrible, horrible interface. At least you can search grants. 
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Design 

This design if horrible. I understand the basic idea of having the tier or tree approach for each dept .. but no 
user is going to understand what those numbers are underneath them. Obviously this data is in some kind 
of database already, so you’ve got to come up with a better way of using the info returned from the db. 
Navigation 

The navigation is not consistent The **back to navigator” button takes me some place I didn’t start from. 
When you expand a “tree”, the page leaps around like crazy and some users won’t realize what happened. 
There are no instructions, either. 

Quality 

No problems here .. all the info is there, like 1 said before - it’s the presentation that’s flawed. 

Customer Service 

Nodiing at all .. it’s very easy to get lost on this site .. and who do you email/call if you have a question? In 
the individ grants themselves there is contact info, but not for die whole site. 



Office of Migrant Education 

One evaluator evaluated the Office of Migrant Education 
Orientation 

Who is the site targeted for? Migrant wodcers themselves? 

Design 

consistent, simple, not distracting. Defiiitely does not suffer from ‘over design’. Could be more visually 
appealing, with color used throughout the pages as opposed to mastheads and text Some inconsistency 
when moving around (Background Colors change for no reason?). Some pages are too long and should be 
separated into several pages (but also present one long version of the page that can easilly be printed see: 
ht^://www.migranted.org/ccdrep.htm) 

Navigation 

Some problems with navigation re: the ‘up’ button which brings you to the top of a section different from 
the one 

You originally want ‘down’ in re: 

The transition from: http://www.ed.gov/ofnces/OESE/MEP/grants.htxnl (click on subgrants) -> 
http://www.ed.gov/ofinces/OESE/MEP/PrelimGuide/pt2e.html and then clicking on the up (should bring 
you back to the previous page, but brings you elsewhere..) 

How about a search function? 

Quality 

information seems up to date, and this can be determined by the ‘freshness dating’ at the bottom of most of 
the pages. No dead-end links or out of date pages were found. 

Customer Service 

No obvious or easy mechanisms to provide general feedback. One mechanism to send in information about 
events, but this is woefully inadequate. Need a more formal way than an easily overlooked mailto: at the 
bottom of most pages. 

Safe and Drug Free Schools 

One evaluator completed the Safe and Drug Free School subsection evaluation 

Orientation: 

<&o comments> 

Design 

Poor organization. Items on the news page that would be considered publications not referenced from the 
publications page. Similar conunents apply to those above, (comments above: consistent, simple, not 
distracting. Defnitely does not suffer from ‘over design’. Could be more visually appealing, with color used 
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throughout the pages as opposed to mastheads and text. Some inconsistency when moving around 
(Background Colors change for no reason?). Some pages are too long and should be separated into several 
pages (but also present one long version of the page that can easilly be printed see: 
http://www.migranted.org/ccdrep.htm)) 

Navigation 

Some links bring you to other sites without hinting at the fact you’ll go there (ie: the link from the 
homepage to the publications ordering site.) 

Quality 

Information seems current, dates at the bottom of the pages are good. 

Customer Service 

mailto: links to feedback, same comment above applies, (comments above: No obvious or easy 
mechanisms to provide general feedback. One mechanism to send in information about events, but diis is 
woefully inadequate. Need a more formal way than an easily overlooked mailto: at die bottom of most 
pages) 



School Improvement 

Two evaluators examined the School Improvement subsection. Their comments follow. 

Orientation: 

<no comments> 

Design 

More visually appealing than the previous pages. Nice colors and design. Not the stale masthead/text 
approach. Lack of consistency as the site is explored however needs to be addressed. 

Navigation 

Straightforward, but there are many links to to other sites, where there’s no way to get you back to the 
Scool Improvement page. 

Quality 

A bit out of date (Not as current as the previous sites) dates are mentioned though. This is good. 

Customer Service 

Insufficient feedback mechanisims. See comments above, (comments above: No obvious or easy 
mechanisms to provide general feedback. One mechanism to send in information about events, but diis is 
woefully inadequate. Need a more formal way than an easily overlooked mailto: at the bottom of most 
pages) 

Orientation 

The splash page provides more in depdi info about the programs administered than the About page; perhaps 
what’s on the splash page should go under the main site. The page describing the programs 
(htqi://www.ed.gov/offlces/OESE/SIP/programs/) is quite good. 

Design 

Some conhision on links to Charter & Magnet Schools. Both main page 
(http://www.ed.gov/offices/OESE/SIP/) & About page 

(http://www.ed.gov/offices/OESE/SIP/aboutsip.html) link to Charter schools site, which is probably 
uimecessarily redundant About page links to Magnet Schools Assistance Programs 
(http://www.ed.gov/offlces/OESE/SIP/itmovmgnJitml), but main page links to Magnet Schools Assistance 
Grant Program, which in turn links to outside sites. These two Magnet schools pages should probably be 
merged in to one. 

Navigation 

Quite simple. 

Did not expect to have to sign a guesAook to look at Region 1 . (http://www.edc.org/NECAC/) Filling out a 

survey should be an option, not a requirement 

Quality 

Didin’t notice any dead links. 
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Customer Service 

Didn’t notice any feedback or help mechanisms, but the site was so small that it wasn’t necessary. 



Main homepage and Associated Links 

Two evaluators examined the OESE homepage. 

Orientation 

Mission and Responsibilities are clear & easy to find. Could use a link on the About Us page to the 
Programs & Fimding, maybe as an indicator of the Scope of OESE. 

Also, good idea to have links to staff members’ individual homepages, though having Imks to their photos 
is probably unnecessary. 

Design 

On the Current Question page (http://www.ed.gov/ofiices/OESE/ESEA/question.html), the responses 
glin ylH be posted as well as the question, to get some discussion going. Maybe have this be a threaded 
discussion list sort of thing. 

Navigation 

Simple to navigate. There are few links per page, making them easy to see. Also, good headings for blocks 
of text Easy on the eye. 

Quality , 

News page is slightly out of date - latest entry is over a week old. And why have an archive of news that s 

over a year old? 

T .tnif to LeamNet on Hot Topics page (http://www.ed.gov/ofBces/OESE/hot.html) is out of date. 

T inlf to National Exchange Carrier Association on same page is dead. 

Customer Service 

P/i ntgp t pagf»,~hiit the link to it maybe should go at die top of the About Us page. Also, why have a 
Questions & Comments page separate from the Contact info page? These 2 pages could easily be merged. 

Orientation: 

<Qo comment> 

Design 

Seems to follow (or establish) a template for the other sites in this section. Unfortunately, the template it 
establishes is dry and isn’t visually stimulating. 

Navigation 

Attempts to be straightforward, easy to get lost 

Quality 

<no comment> 

Customer Service 

See rninTnmtg on feedback above. Need something more than a mailto: link! (No obvious or easy 
.TWK»iignicmg to provide general feedback. One mechanism to send in information about events, but this is 
woefully inadequate. Need a more formal way than an easily overlooked mailto: at the bottom of most 
pages) 
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